System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface

ABSTRACT

A system and method for configuring the presentation of a plurality of presentation elements in a visual representation on a user interface, the presentation elements having both temporal and spatial parameters, the method comprising the steps of: defining a time bar with a time scale having time indicators as subdivisions of the time scale and having a first global temporal limit and a second temporal global limit of the time scale for defining a temporal domain of the presentation elements, defining a focus range of the time bar such that the focus range has a first local temporal limit and a second local temporal limit wherein the first local temporal limit is greater than or equal to the first global temporal limit and the second local temporal limit is less than or equal to the second global temporal limit; defining a focus bar having a focus time scale having focus time indicators as subdivisions of the focus time scale and having the first and second local temporal limits as the extents of the focus time scale, such that the focus time scale is an expansion of the time scale; and displaying a set of presentation elements selected from the plurality of presentation elements based on the respective temporal parameter of each of the set of presentation elements is within the first and second local temporal limits.

CROSS-REFERENCE TO RELATED APPLICATION PARAGRAPH

This application claims the benefit of Canadian Patent Application No.2,646,117, filed Dec. 2, 2008 and claims the benefit of U.S. ProvisionalApplication No. 61/193,495, filed on Dec. 3, 2008. The contents of bothapplications are incorporated herein in their entirety by reference.

BACKGROUND OF THE INVENTION

The present invention relates to an interactive visual presentation ofmultidimensional data on a user interface.

Tracking and analyzing entities and streams of events, has traditionallybeen the domain of investigators, whether that be national intelligenceanalysts, police services or military intelligence. Business users alsoanalyze events in time and location to better understand phenomenon suchas customer behavior or transportation patterns. As data about eventsand objects become more commonly available, analyzing and understandingof interrelated temporal and spatial information is increasingly aconcern for military commanders, intelligence analysts and businessanalysts. Localized cultures, characters, organizations and theirbehaviors play an important part in planning and mission execution. Insituations of asymmetric warfare and peacekeeping, tracking relativelysmall and seemingly unconnected events over time becomes a means fortracking enemy behavior. For business applications, tracking ofproduction process characteristics can be a means for improving plantoperations. A generalized method to capture and visualize thisinformation over time for use by business and military applications,among others, is needed.

Many visualization techniques and products for analyzing complex eventinteractions only display information along a single dimension,typically one of time, geography or a network connectivity diagram. Eachof these types of visualizations is common and well understood. Forexample a Time-focused scheduling chart such as Microsoft (MS) Projectdisplays various project events over the single dimension of time, and aGeographic Information System (GIS) product, such as MS MapPoint, orESRI ArcView, is good for showing events in the single dimension oflocations on a map. There are also link analysis tools, such as Netmap(www.netmapanalytics.com) or Visual Analytics (www.visualanalytics.com)that display events as a network diagram, or graph, of objects andconnections between objects. Some of these systems are capable of usinganimation to display another dimension, typically time. Time is playedback, or scrolled, and the related spatial image display changes toreflect the state of information at a moment in time. However thistechnique relies on limited human short term memory to track and thenretain temporal changes and patterns in the spatial domain. Anothervisualization technique called “small multiples” uses repeated frames ofa condition or chart, each capturing an increment moment in time, muchlike looking at sequence of frames from a film laid side by side. Eachimage must be interpreted separately, and side-by-side comparisons made,to detect differences. This technique is expensive in terms of visualspace since an image must be generated for each moment of interest,which can be problematic when trying to simultaneously display multipleimages of adequate size that contain complex data content.

A technique has been developed, as described in InteractiveVisualization of Spatiotemporal Patterns using Spirals on a GeographicalMap—by Hewagamage et al. that uses spiral shaped ribbons as timelines toshow isolated sequences of events that have occurred at discretelocations on a geographical map. This technique is limited because ituses spiral timelines exclusively to show the periodic quality ofcertain types of events, while does not show connectivity between thetemporal and spatial information of data objects at multi-locationswithin the spatial domain. Further, event data objects placed on thespirals can suffer from occlusion, thereby providing for only a limitednumber of events and locations viewable with the spiral timelines.

Further, the ability for current visualization systems to sortefficiently (i.e. selective visual display of elements) using temporalproperties of the presentation elements is lacking

SUMMARY

It is an object of the present invention to provide a system and methodfor the interactive visual representation of a plurality of presentationobjects with spatial and temporal properties to obviate or mitigate atleast some of the above-mentioned disadvantages.

A first aspect provided is a method for configuring the presentation ofa plurality of presentation elements in a visual representation on auser interface, the presentation elements having both temporal andspatial parameters, the method comprising the steps of: defining a timebar with a time scale having time indicators as subdivisions of the timescale and having a first global temporal limit and a second temporalglobal limit of the time scale for defining a temporal domain of thepresentation elements, defining a focus range of the time bar such thatthe focus range has a first local temporal limit and a second localtemporal limit wherein the first local temporal limit is greater than orequal to the first global temporal limit and the second local temporallimit is less than or equal to the second global temporal limit;defining a focus bar having a focus time scale having focus timeindicators as subdivisions of the focus time scale and having the firstand second local temporal limits as the extents of the focus time scale,such that the focus time scale is an expansion of the time scale; anddisplaying a set of presentation elements selected from the plurality ofpresentation elements based on the respective temporal parameter of eachof the set of presentation elements is within the first and second localtemporal limits.

A second aspect provided is a system for configuring the presentation ofa plurality of presentation elements in a visual representation on auser interface, the presentation elements having both temporal andspatial parameters, the system comprising: a time module configured fordefining a time bar with a time scale having time indicators assubdivisions of the time scale and having a first global temporal limitand a second temporal global limit of the time scale for defining atemporal domain of the presentation elements, and for defining a focusrange of the time bar such that the focus range has a first localtemporal limit and a second local temporal limit wherein the first localtemporal limit is greater than or equal to the first global temporallimit and the second local temporal limit is less than or equal to thesecond global temporal limit; a focus module configured for defining afocus bar having a focus time scale having focus time indicators assubdivisions of the focus time scale and having the first and secondlocal temporal limits as the extents of the focus time scale, such thatthe focus time scale is an expansion of the time scale; and a visualmodule configured for displaying a set of presentation elements selectedfrom the plurality of presentation elements based on the respectivetemporal parameter of each of the set of presentation elements is withinthe first and second local temporal limits.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of these and other embodiments of the presentinvention can be obtained with reference to the following drawings anddetailed description of the preferred embodiments, in which:

FIG. 1 is a block diagram of a data processing system for avisualization tool;

FIG. 2 shows further details of the data processing system of FIG. 1;

FIG. 3 shows further details of the visualization tool of FIG. 1;

FIG. 4 shows further details of a visualization representation fordisplay on a visualization interface of the system of FIG. 1;

FIG. 5 is an example visualization representation of FIG. 1 showingEvents in Concurrent Time and Space;

FIG. 6 shows example data objects and associations of FIG. 1;

FIG. 7 shows further example data objects and associations of FIG. 1;

FIG. 8 shows changes in orientation of a reference surface of thevisualization representation of FIG. 1;

FIG. 9 is an example timeline of FIG. 8;

FIG. 10 is a further example timeline of FIG. 8;

FIG. 11 is a further example timeline of FIG. 8 showing a time chart;

FIG. 12 is a further example of the time chart of FIG. 11;

FIG. 13 shows example user controls for the visualization representationof FIG. 5;

FIG. 14 shows an example operation of the tool of FIG. 3;

FIG. 15 shows a further example operation of the tool of FIG. 3;

FIG. 16 shows a further example operation of the tool of FIG. 3;

FIG. 17 shows an example visualization representation of FIG. 4containing events and target tracking over space and time showingconnections between events;

FIG. 18 shows an example visualization representation containing eventsand target tracking over space and time showing connections betweenevents on a time chart of FIG. 11, and

FIG. 19 is an example operation of the visualization tool of FIG. 3;

FIG. 20 is a further embodiment of FIG. 18 showing imagery;

FIG. 21 is a further embodiment of FIG. 18 showing imagery in a timechart view;

FIG. 22 shows further detail of the aggregation module of FIG. 3;

FIG. 23 shows an example aggregation result of the module of FIG. 22;

FIG. 24 is a further embodiment of the result of FIG. 23;

FIG. 25 shows a summary chart view of a further embodiment of therepresentation of FIG. 20;

FIG. 26 shows an event comparison for the aggregation module of FIG. 23;

FIG. 27 shows a further embodiment of the tool of FIG. 3;

FIG. 28 shows an example operation of the tool of FIG. 27;

FIG. 29 shows a further example of the visualization representation ofFIG. 4;

FIG. 30 is a further example of the charts of FIG. 25;

FIGS. 31 a,b,c,d show example control sliders of analysis functions ofthe tool of FIG. 3;

FIG. 32 shows a visualization tool for generating stories in the timeand space domains;

FIG. 33 shows an example of the visualization representation of FIG. 32;

FIG. 34 shows an example visualization representation prior to analysisby the visualization tool of FIG. 32;

FIG. 35 shows an example aggregation result of the module of FIG. 32;

FIG. 36 shows an example aggregation and pattern matching analysisapplied to FIG. 35;

FIGS. 37 a,b show example generation of a story element of a story ofFIG. 32;

FIG. 38 shows an exemplary process for processing data objects for anexisting story using the visualization tool of FIG. 32;

FIG. 39 is an embodiment of a pattern template for generating the storyelements of FIG. 32;

FIG. 40 is a further embodiment of the visualization representation ofFIG. 32;

FIG. 41 is a further embodiment of the visualization representation ofFIG. 32;

FIG. 42 is a further embodiment of the visualization representation ofFIG. 32;

FIG. 43 is an example story framework generated using the text module ofFIG. 32;

FIG. 44 shows an example operation for generating the story framework ofFIG. 43; and

FIG. 45 is a further embodiment of generating the story element forFIGS. 37 a,b;

FIGS. 46 a,b,c,d,e,f show example operations of the timeline bar andfocus bar of the tool of FIG. 3;

FIG. 46 g is an embodiment of visualization of the tool of FIG. 1;

FIG. 47 shows a further embodiment of the tool of FIG. 3;

FIG. 48 illustrates a further embodiment of the tool of FIG. 3;

FIGS. 49 a,b,c,d,e,f,g,h show example analysis tool controls of the toolof FIG. 48;

FIGS. 50 a,b show a further embodiment of the tool of FIG. 3;

FIG. 51 shows a further embodiment of the tool of FIG. 3;

FIG. 52 shows an example operation of a count chart control of the toolof FIG. 51;

FIG. 53 shows an example operation of a callout annotation of the toolof FIG. 3;

FIG. 54 shows an example operation of a chart annotation of the tool ofFIG. 3;

FIG. 55 shows an example operation of a group annotation of the tool ofFIG. 3;

FIG. 56 shows an example operation of a line annotation of the tool ofFIG. 3;

FIG. 57 shows an example operation of a ruler annotation of the tool ofFIG. 3; and

FIG. 58 shows an example operation of a symbols annotation of the toolof FIG. 3.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The following detailed description of the embodiments of the presentinvention does not limit the implementation of the invention to anyparticular computer programming language. The present invention may beimplemented in any computer programming language provided that the OS(Operating System) provides the facilities that may support therequirements of the present invention. A preferred embodiment isimplemented in the Java computer programming language (or other computerprogramming languages in conjunction with C/C++). Any limitationspresented would be a result of a particular type of operating system,computer programming language, or data processing system and would notbe a limitation of the present invention.

Visualization Environment

Referring to FIG. 1, a visualization data processing system 100 includesa visualization tool 12 for processing a collection of data objects 14as input data elements to a user interface 202. The data objects 14 arecombined with a respective set of associations 16 by the tool 12 togenerate an interactive visual representation 18 on the visual interface(VI) 202. The data objects 14 include event objects 20, location objects22, images 23 and entity objects 24, as further described below. The setof associations 16 include individual associations 26 that associatetogether various subsets of the objects 20, 22, 23, 24, as furtherdescribed below. Management of the data objects 14 and set ofassociations 16 are driven by user events 109 of a user (not shown) viathe user interface 108 (see FIG. 2) during interaction with the visualrepresentation 18. The representation 18 shows connectivity betweentemporal and spatial information of data objects 14 at multi-locationswithin the spatial domain 400 (see FIG. 4).

Data Processing System 100

Referring to FIG. 2, the data processing system 100 has a user interface108 for interacting with the tool 12, the user interface 108 beingconnected to a memory 102 via a BUS 106. The interface 108 is coupled toa processor 104 via the BUS 106, to interact with user events 109 tomonitor or otherwise instruct the operation of the tool 12 via anoperating system 110. The user interface 108 can include one or moreuser input devices such as but not limited to a QWERTY keyboard, akeypad, a trackwheel, a stylus, a mouse, and a microphone. The visualinterface 202 is considered the user output device, such as but notlimited to a computer screen display. If the screen is touch sensitive,then the display can also be used as the user input device as controlledby the processor 104. The operation of the data processing system 100 isfacilitated by the device infrastructure including one or more computerprocessors 104 and can include the memory 102 (e.g. a random accessmemory). The computer processor(s) 104 facilitates performance of thedata processing system 100 configured for the intended task(s) throughoperation of a network interface, the user interface 202 and otherapplication programs/hardware of the data processing system 100 byexecuting task related instructions. These task related instructions canbe provided by an operating system, and/or software applications locatedin the memory 102, and/or by operability that is configured into theelectronic/digital circuitry of the processor(s) 104 designed to performthe specific task(s).

Further, it is recognized that the data processing system 100 caninclude a computer readable storage medium 46 coupled to the processor104 for providing instructions to the processor 104 and/or the tool 12.The computer readable medium 46 can include hardware and/or softwaresuch as, by way of example only, magnetic disks, magnetic tape,optically readable medium such as CD/DVD ROMS, and memory cards. In eachcase, the computer readable medium 46 may take the form of a small disk,floppy diskette, cassette, hard disk drive, solid-state memory card, orRAM provided in the memory 102. It should be noted that the above listedexample computer readable mediums 46 can be used either alone or incombination.

Referring again to FIG. 2, the tool 12 interacts via link 116 with a VImanager 112 (also known as a visualization renderer) of the system 100for presenting the visual representation 18 on the visual interface 202.The tool 12 also interacts via link 118 with a data manager 114 of thesystem 100 to coordinate management of the data objects 14 andassociation set 16 from data files or tables 122 of the memory 102. Itis recognized that the objects 14 and association set 16 could be storedin the same or separate tables 122, as desired. The data manager 114 canreceive requests for storing, retrieving, amending, or creating theobjects 14 and association set 16 via the tool 12 and/or directly vialink 120 from the VI manager 112, as driven by the user events 109and/or independent operation of the tool 12. The data manager 114manages the objects 14 and association set 16 via link 123 with thetables 122. Accordingly, the tool 12 and managers 112, 114 coordinatethe processing of data objects 14, association set 16 and user events109 with respect to the content of the screen representation 18displayed in the visual interface 202.

The task related instructions can comprise code and/or machine readableinstructions for implementing predetermined functions/operationsincluding those of an operating system, tool 12, or other informationprocessing system, for example, in response to command or input providedby a user of the system 100. The processor 104 (also referred to asmodule(s) for specific components of the tool 12) as used herein is aconfigured device and/or set of machine-readable instructions forperforming operations as described by example above.

As used herein, the processor/modules in general may comprise any one orcombination of, hardware, firmware, and/or software. Theprocessor/modules acts upon information by manipulating, analyzing,modifying, converting or transmitting information for use by anexecutable procedure or an information device, and/or by routing theinformation with respect to an output device. The processor/modules mayuse or comprise the capabilities of a controller or microprocessor, forexample. Accordingly, any of the functionality provided by the systemsand process of FIGS. 1-45 may be implemented in hardware, software or acombination of both. Accordingly, the use of a processor/modules as adevice and/or as a set of machine readable instructions is hereafterreferred to generically as a processor/module for sake of simplicity.

It will be understood by a person skilled in the art that the memory 102storage described herein is the place where data is held in anelectromagnetic or optical form for access by a computer processor. Inone embodiment, storage means the devices and data connected to thecomputer through input/output operations such as hard disk and tapesystems and other forms of storage not including computer memory andother in-computer storage. In a second embodiment, in a more formalusage, storage is divided into: (1) primary storage, which holds data inmemory (sometimes called random access memory or RAM) and other“built-in” devices such as the processor's L1 cache, and (2) secondarystorage, which holds data on hard disks, tapes, and other devicesrequiring input/output operations. Primary storage can be much faster toaccess than secondary storage because of the proximity of the storage tothe processor or because of the nature of the storage devices. On theother hand, secondary storage can hold much more data than primarystorage. In addition to RAM, primary storage includes read-only memory(ROM) and L1 and L2 cache memory. In addition to hard disks, secondarystorage includes a range of device types and technologies, includingdiskettes, Zip drives, redundant array of independent disks (RAID)systems, and holographic storage. Devices that hold storage arecollectively known as storage media.

A database is a further embodiment of memory 102 as a collection ofinformation that is organized so that it can easily be accessed,managed, and updated. In one view, databases can be classified accordingto types of content: bibliographic, full-text, numeric, and images. Incomputing, databases are sometimes classified according to theirorganizational approach. As well, a relational database is a tabulardatabase in which data is defined so that it can be reorganized andaccessed in a number of different ways. A distributed database is onethat can be dispersed or replicated among different points in a network.An object-oriented programming database is one that is congruent withthe data defined in object classes and subclasses.

Computer databases typically contain aggregations of data records orfiles, such as sales transactions, product catalogs and inventories, andcustomer profiles. Typically, a database manager provides users thecapabilities of controlling read/write access, specifying reportgeneration, and analyzing usage. Databases and database managers areprevalent in large mainframe systems, but are also present in smallerdistributed workstation and mid-range systems such as the AS/400 and onpersonal computers. SQL (Structured Query Language) is a standardlanguage for making interactive queries from and updating a databasesuch as IBM's DB2, Microsoft's Access, and database products fromOracle, Sybase, and Computer Associates.

Memory is a further embodiment of memory 210 storage as the electronicholding place for instructions and data that the computer'smicroprocessor can reach quickly. When the computer is in normaloperation, its memory usually contains the main parts of the operatingsystem and some or all of the application programs and related data thatare being used. Memory is often used as a shorter synonym for randomaccess memory (RAM). This kind of memory is located on one or moremicrochips that are physically close to the microprocessor in thecomputer.

Referring to FIGS. 27 and 29, the tool 12 can have an information module712 for generating information 714 a,b,c,d for display by thevisualization manager 300, in response to user manipulations via the I/Ointerface 108. For example, when a mouse pointer 713 is held over thevisual element 410,412 of the representation 18, some predefinedinformation 714 a,b,c,d is displayed about that selected visual element410,412. The information module 712 is configured to display the type ofinformation dependent upon whether the object is a place 22, target 24,elementary or compound event 20, for example. For example, when theplace 22 type is selected, the displayed information 714 a is formattedby the information module 712 to include such as but not limited to;Label (e.g. Rome), Attributes attached to the object (if any); andevents associated with that place 22. For example, when the target24/target trail 412 (see FIG. 17) type is selected, the displayedinformation 714 b is formatted by the information module 712 to includesuch as but not limited to; Label, Attributes (if any), eventsassociated with that target 24, as well as the target's icon (if one isassociated with the target 24) is shown. For example, when an elementaryevent 20 a type is selected, the displayed information 714 c isformatted by the information module 712 to include such as but notlimited to; Label, Class, Date, Type, Comment (including Attributes, ifany), associated Targets 24 and Place 22. For example, when a compoundevent 20 b type is selected, the displayed information 714 d isformatted by the information module 712 to include such as but notlimited to; Label, Class, Date, Type, Comment (including Attributes, ifany) and all elementary event popup data for each child event.Accordingly, it is recognized that the information module 712 isconfigured to select data for display from the database 122 (see FIG. 2)appropriate to the type of visual element 410,412 selected by the userfrom the visual representation 18.

Tool Information Model

Referring to FIG. 1, a tool information model is composed of the fourbasic data elements (objects 20, 22, 23, 24 and associations 26) thatcan have corresponding display elements in the visual representation 18.The four elements are used by the tool 12 to describe interconnectedactivities and information in time and space as the integrated visualrepresentation 18, as further described below.

Event Data Objects 20

Events are data objects 20 that represent any action that can bedescribed. The following are examples of events;

-   -   Bill was at Toms house at 3 pm,    -   Tom phoned Bill on Thursday,    -   A tree fell in the forest at 4:13 am, Jun. 3, 1993 and    -   Tom will move to Spain in the summer of 2004.        The Event is related to a location and a time at which the        action took place, as well as several data properties and        display properties including such as but not limited to; a short        text label, description, location, start-time, end-time, general        event type, icon reference, visual layer settings, priority,        status, user comment, certainty value, source of information,        and default+user-set color. The event data object 20 can also        reference files such as images or word documents.

Locations and times may be described with varying precision. Forexample, event times can be described as “during the week of January5^(th)” or “in the month of September”. Locations can be described as“Spain” or as “New York” or as a specific latitude and longitude.

Entity Data Objects 24

Entities are data objects 24 that represent any thing related to orinvolved in an event, including such as but not limited to; people,objects, organizations, equipment, businesses, observers, affiliationsetc. Data included as part of the Entity data object 24 can be shorttext label, description, general entity type, icon reference, visuallayer settings, priority, status, user comment, certainty value, sourceof information, and default+user-set color. The entity data can alsoreference files such as images or word documents. It is recognized inreference to FIGS. 6 and 7 that the term Entities includes “People”, aswell as equipment (e.g. vehicles), an entire organization (e.g.corporate entity), currency, and any other object that can be trackedfor movement in the spatial domain 400. It is also recognized that theentities 24 could be stationary objects such as but not limited tobuildings. Further, entities can be phone numbers and web sites. To beexplicit, the entities 24 as given above by example only can be regardedas Actors

Location Data Objects 22

Locations are data objects 22 that represent a place within a spatialcontext/domain, such as a geospatial map, a node in a diagram such as aflowchart, or even a conceptual place such as “Shang-ri-la” or other“locations” that cannot be placed at a specific physical location on amap or other spatial domain. Each Location data object 22 can store suchas but not limited to; position coordinates, a label, description, colorinformation, precision information, location type, non-geospatial flagand user comments.

Associations

Event 20, Location 22 and Entity 24 are combined into groups or subsetsof the data objects 14 in the memory 102 (see FIG. 2) using associations26 to describe real-world occurrences. The association is defined as aninformation object that describes a pairing between 2 data objects 14.For example, in order to show that a particular entity was present whenan event occurred, the corresponding association 26 is created torepresent that Entity X “was present at” Event A. For example,associations 26 can include such as but not limited to; describing acommunication connection between two entities 24, describing a physicalmovement connection between two locations of an entity 24, and arelationship connection between a pair of entities 24 (e.g. familyrelated and/or organizational related). It is recognised that theassociations 26 can describe direct and indirect connections. Otherexamples can include phone numbers and web sites.

A variation of the association type 26 can be used to define a subclassof the groups 27 to represent user hypotheses. In other words, groups 27can be created to represent a guess or hypothesis that an eventoccurred, that it occurred at a certain location or involved certainentities. Currently, the degree of belief/accuracy/evidence reliabilitycan be modeled on a simple 1-2-3 scale and represented graphically withline quality on the visual representation 18.

Image Data Objects 23

Standard icons for data objects 14 as well as small images 23 for suchas but not limited to objects 20,22,24 can be used to describe entitiessuch as people, organizations and objects. Icons are also used todescribe activities. These can be standard or tailored icons, or actualimages of people, places, and/or actual objects (e.g. buildings).Imagery can be used as part of the event description. Images 23 can beviewed in all of the visual representation 18 contexts, as for exampleshown in FIGS. 20 and 21, which show the use of images 23 in the timelines 422 and the time chart 430 views. Sequences of images 23 can beanimated to help the user detect changes in the image over time andspace.

Annotations 21

Annotations 21 in Geography and Time (see FIG. 22) can be represented asmanually placed lines or other shapes (e.g. pen/pencil strokes) can beplaced on the visual representation 18 by an operator of the tool 12 andused to annotate elements of interest with such as but not limited toarrows, circles and freeform markings. Some examples are shown in FIG.21. These annotations 21 are located in geography (e.g. spatial domain400) and time (e.g. temporal domain 422) and so can appear and disappearon the visual representation 18 as geographic and time contexts arenavigated through the user input events 109.

Visualization Tool 12

Referring to FIG. 3, the visualization tool 12 has a visualizationmanager 300 for interacting with the data objects 14 for presentation tothe interface 202 via the VI manager 112. The Data Objects 14 are formedinto groups 27 through the associations 26 and processed by theVisualization Manager 300. The groups 27 comprise selected subsets ofthe objects 20, 21, 22, 23, 24 combined via selected associations 26.This combination of data objects 14 and association sets 16 can beaccomplished through predefined groups 27 added to the tables 122 and/orthrough the user events 109 during interaction of the user directly withselected data objects 14 and association sets 16 via the controls 306.It is recognized that the predefined groups 27 could be loaded into thememory 102 (and tables 122) via the computer readable medium 46 (seeFIG. 2). The Visualization manager 300 also processes user event 109input through interaction with a time slider and other controls 306,including several interactive controls for supporting navigation andanalysis of information within the visual representation 18 (see FIG. 1)such as but not limited to data interactions of selection, filtering,hide/show and grouping as further described below. Use of the groups 27is such that subsets of the objects 14 can be selected and groupedthrough associations 26. In this way, the user of the tool 12 canorganize observations into related stories or story fragments. Thesegroupings 27 can be named with a label and visibility controls, whichprovide for selected display of the groups 27 on the representation 18,e.g. the groups 27 can be turned on and off with respect to display tothe user of the tool 12.

The Visualization Manager 300 processes the translation from raw dataobjects 14 to the visual representation 18. First, Data Objects 14 andassociations 16 can be formed by the Visualization Manager 300 into thegroups 27, as noted in the tables 122, and then processed. TheVisualization Manager 300 matches the raw data objects 14 andassociations 16 with sprites 308 (i.e. visual processingobjects/components that know how to draw and render visual elements forspecified data objects 14 and associations 16) and sets a drawingsequence for implementation by the VI manager 112. The sprites 308 arevisualization components that take predetermined information schema asinput and output graphical elements such as lines, text, images andicons to the computers graphics system. Entity 24, event 20 and location22 data objects each can have a specialized sprite 308 type designed torepresent them. A new sprite instance is created for each entity, eventand location instance to manage their representation in the visualrepresentation 18 on the display.

The sprites 308 are processed in order by the visualization manager 300,starting with the spatial domain (terrain) context and locations,followed by Events and Timelines, and finally Entities. Timelines aregenerated and Events positioned along them. Entities are rendered lastby the sprites 308 since the entities depend on Event positions. It isrecognised that processing order of the sprites 308 can be other than asdescribed above.

The Visualization manager 112 renders the sprites 308 to create thefinal image including visual elements representing the data objects 14and associates 16 of the groups 27, for display as the visualrepresentation 18 on the interface 202. After the visual representation18 is on the interface 202, the user event 109 inputs flow into theVisualization Manager, through the VI manager 112 and cause the visualrepresentation 18 to be updated. The Visualization Manager 300 can beoptimized to update only those sprites 308 that have changed in order tomaximize interactive performance between the user and the interface 202.

Layout of the Visualization Representation 18

The visualization technique of the visualization tool 12 is designed toimprove perception of entity activities, movements and relationships asthey change over time in a concurrent time-geographic ortime-diagrammatical context. The visual representation 18 of the dataobjects 14 and associations 16 consists of a combined temporal-spatialdisplay to show interconnecting streams of events over a range of timeon a map or other schematic diagram space, both hereafter referred to incommon as a spatial domain 400 (see FIG. 4). Events can be representedwithin an X,Y,T coordinate space, in which the X,Y plane shows thespatial domain 400 (e.g. geographic space) and the Z-axis represents atime series into the future and past, referred to as a temporal domain402. In addition to providing the spatial context, a reference surface(or reference spatial domain) 404 marks an instant of focus betweenbefore and after, such that events “occur” when they meet the surface ofthe ground reference surface 404. FIG. 4 shows how the visualizationmanager 300 (see FIG. 3) combines individual frames 406 (spatial domains400 taken at different times Ti 407) of event/entity/location visualelements 410, which are translated into a continuous integrated spatialand temporal visual representation 18. It should be noted connectionvisual elements 412 can represent presumed location (interpolated) ofEntity between the discrete event/entity/location represented by thevisual elements 410. Another interpretation for connections elements 412could be signifying communications between different Entities atdifferent locations, which are related to the same event as furtherdescribed below.

Referring to FIG. 5, an example visual representation 18 visuallydepicts events over time and space in an x, y, t space (or x, y, z, tspace with elevation data). The example visual representation 18generated by the tool 12 (see FIG. 2) is shown having the time domain402 as days in April, and the spatial domain 400 as a geographical mapproviding the instant of focus (of the reference surface 404) assometime around noon on April 23—the intersection point between thetimelines 422 and the reference surface 404 represents the instant offocus. The visualization representation 18 represents the temporal 402,spatial 400 and connectivity elements 412 (between two visual elements410) of information within a single integrated picture on the interface202 (see FIG. 1). Further, the tool 12 provides an interactive analysistool for the user with interface controls 306 to navigate the temporal,spatial and connectivity dimensions. The tool 12 is suited to theinterpretation of any information in which time, location andconnectivity are key dimensions that are interpreted together. Thevisual representation 18 is used as a visualization technique fordisplaying and tracking events, people, and equipment within thecombined temporal and spatial domains 402, 400 display. Tracking andanalyzing entities 24 and streams has traditionally been the domain ofinvestigators, whether that be police services or military intelligence.In addition, business users also analyze events 20 in time and spatialdomains 400, 402 to better understand phenomenon such as customerbehavior or transportation patterns. The visualization tool 12 can beapplied for both reporting and analysis.

The visual representation 18 can be applied as an analyst workspace forexploration, deep analysis and presentation for such as but not limitedto:

-   -   Situations involving people and organizations that interact over        time and in which geography or territory plays a role;    -   Storing and reviewing activity reports over a given period. Used        in this way the representation 18 could provide a means to        determine a living history, context and lessons learned from        past events; and    -   As an analysis and presentation tool for long term tracking and        surveillance of persons and equipment activities.

The visualization tool 12 provides the visualization representation 18as an interactive display, such that the users (e.g. intelligenceanalysts, business marketing analysts) can view, and work with, largenumbers of events. Further, perceived patterns, anomalies andconnections can be explored and subsets of events can be grouped into“story” or hypothesis fragments. The visualization tool 12 includes avariety of capabilities such as but not limited to:

-   -   An event-based information architecture with places, events,        entities (e.g. people) and relationships;    -   Past and future time visibility and animation controls;    -   Data input wizards for describing single events and for loading        many events from a table;    -   Entity and event connectivity analysis in time and geography;    -   Path displays in time and geography;    -   Configurable workspaces allowing ad hoc, drag and drop        arrangements of events;    -   Search, filter and drill down tools;    -   Creation of sub-groups and overlays by selecting events and        dragging them into sets (along with associated spatial/time        scope properties); and    -   Adaptable display functions including dynamic show/hide        controls.        Example Objects 14 with Associations 16

In the visualization tool 12, specific combinations of associated dataelements (objects 20, 22, 24 and associations 26) can be defined. Thesedefined groups 27 are represented visually as visual elements 410 inspecific ways to express various types of occurrences in the visualrepresentation 18. The following are examples of how the groups 27 ofassociated data elements can be formed to express specific occurrencesand relationships shown as the connection visual elements 412.

Referring to FIGS. 6 and 7, example groups 27 (denoting common realworld occurrences) are shown with selected subsets of the objects 20,22, 24 combined via selected associations 26. The correspondingvisualization representation 18 is shown as well including the temporaldomain 402, the spatial domain 400, connection visual elements 412 andthe visual elements 410 representing the event/entity/locationcombinations. It is noted that example applications of the groups 27 aresuch as but not limited to those shown in FIGS. 6 and 7. In the FIGS. 6and 7 it is noted that event objects 20 are labeled as “Event 1”, “Event2”, location objects 22 are labeled as “Location A”, “Location B”, andentity objects 24 are labeled as “Entity X”, “Entity Y”. The set ofassociations 16 are labeled as individual associations 26 withconnections labeled as either solid or dotted lines 412 between twoevents, or dotted in the case of an indirect connection between twolocations.

Visual Elements Corresponding to Spatial and Temporal Domains

The visual elements 410 and 412, their variations and behaviorfacilitate interpretation of the concurrent display of events in thetime 402 and space 400 domains. In general, events reference thelocation at which they occur and a list of Entities and their role inthe event. The time at which the event occurred or the time span overwhich the event occurred are stored as parameters of the event.

Spatial Domain Representation

Referring to FIG. 8, the primary organizing element of the visualizationrepresentation 18 is the 2D/3D spatial reference frame (subsequentlyincluded herein with reference to the spatial domain 400). The spatialdomain 400 consists of a true 2D/3D graphics reference surface 404 inwhich a 2D or 3 dimensional representation of an area is shown. Thisspatial domain 400 can be manipulated using a pointer device (notshown—part of the controls 306—see FIG. 3) by the user of the interface108 (see FIG. 2) to rotate the reference surface 404 with respect to aviewpoint 420 or viewing ray extending from a viewer 423. The user (i.e.viewer 423) can also navigate the reference surface 404 by scrolling inany direction, zooming in or out of an area and selecting specific areasof focus. In this way the user can specify the spatial dimensions of anarea of interest the reference surface 404 in which to view events intime. The spatial domain 400 represents space essentially as a plane(e.g. reference surface 404), however is capable of representing 3dimensional relief within that plane in order to express geographicalfeatures involving elevation. The spatial domain 400 can be madetransparent so that timelines 422 of the temporal domain 402 can extendbehind the reference surface 404 are still visible to the user. FIG. 8shows how the viewer 423 facing timelines 422 can rotate to face theviewpoint 420 no matter how the reference surface 404 is rotated in 3dimensions with respect to the viewpoint 420.

The spatial domain 400 includes visual elements 410, 412 (see FIG. 4)that can represent such as but not limited to map information, digitalelevation data, diagrams, and images used as the spatial context. Thesetypes of spaces can also be combined into a workspace. The user can alsocreate diagrams using drawing tools (of the controls 306—see FIG. 3)provided by the visualization tool 12 to create custom diagrams andannotations within the spatial domain 400.

Event Representation and Interactions

Referring to FIGS. 4 and 8, events are represented by a glyph, or iconas the visual element 410, placed along the timeline 422 at the point intime that the event occurred. The glyph can be actually a group ofgraphical objects, or layers, each of which expresses the content of theevent data object 20 (see FIG. 1) in a different way. Each layer can betoggled and adjusted by the user on a per event basis, in groups oracross all event instances. The graphical objects or layers for eventvisual elements 410 are such as but not limited to:

1. Text Label

-   -   The Text label is a text graphic meant to contain a short        description of the event content. This text always faces the        viewer 423 no matter how the reference surface 404 is oriented.        The text label incorporates a de-cluttering function that        separates it from other labels if they overlap. When two events        are connected with a line (see connections 412 below) the label        will be positioned at the midpoint of the connection line        between the events. The label will be positioned at the end of a        connection line that is clipped at the edge of the display area.

2. Indicator—Cylinder, Cube or Sphere

-   -   The indicator marks the position in time. The color of the        indicator can be manually set by the user in an event properties        dialog. Color of event can also be set to match the Entity that        is associated with it. The shape of the event can be changed to        represent different aspect of information and can be set by the        user. Typically it is used to represent a dimension such as type        of event or level of importance.

3. Icon

-   -   An icon or image can also be displayed at the event location.        This icon/image 23 may used to describe some aspect of the        content of the event. This icon/image 23 may be user-specified        or entered as part of a data file of the tables 122 (see FIG.        2).

4. Connection Elements 412

-   -   Connection elements 412 can be lines, or other geometrical        curves, which are solid or dashed lines that show connections        from an event to another event, place or target. A connection        element 412 may have a pointer or arrowhead at one end to        indicate a direction of movement, polarity, sequence or other        vector-like property. If the connected object is outside of the        display area, the connection element 412 can be coupled at the        edge of the reference surface 404 and the event label will be        positioned at the clipped end of the connection element 412.

5. Time Range Indicator

-   -   A Time Range Indicator (not shown) appears if an event occurs        over a range of time. The time range can be shown as a line        parallel to the timeline 422 with ticks at the end points. The        event Indicator (see above) preferably always appears at the        start time of the event.

The Event visual element 410 can also be sensitive to interaction. Thefollowing user events 109 via the user interface 108 (see FIG. 2) arepossible, such as but not limited to:

-   Mouse-Left-Click:    -   Selects the visual element 410 of the visualization        representation 18 on the VI 202 (see FIG. 2) and highlights it,        as well as simultaneously deselecting any previously selected        visual element 410, as desired.-   Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click    -   Adds the visual element 410 to an existing selection set.-   Mouse-Left-Double-Click:    -   Opens a file specified in an event data parameter if it exists.        The file will be opened in a system-specified default        application window on the interface 202 based on its file type.-   Mouse-Right-Click:    -   Displays an in-context popup menu with options to hide, delete        and set properties.-   Mouse Over Drilldown:    -   When the mouse pointer (not shown) is placed over the indicator,        a text window is displayed next to the pointer, showing        information about the visual element 410. When the mouse pointer        is moved away from the indicator, the text window disappears.        Location Representation

Locations are visual elements 410 represented by a glyph, or icon,placed on the reference surface 404 at the position specified by thecoordinates in the corresponding location data object 22 (see FIG. 1).The glyph can be a group of graphical objects, or layers, each of whichexpresses the content of the location data object 22 in a different way.Each layer can be toggled and adjusted by the user on a per Locationbasis, in groups or across all instances. The visual elements 410 (e.g.graphical objects or layers) for Locations are such as but not limitedto:

1. Text Label

-   -   The Text label is a graphic object for displaying the name of        the location. This text always faces the viewer 422 no matter        how the reference surface 404 is oriented. The text label        incorporates a de-cluttering function that separates it from        other labels if they overlap.

2. Indicator

-   -   The indicator is an outlined shape that marks the position or        approximate position of the Location data object 22 on the        reference surface 404. There are, such as but not limited to, 7        shapes that can be selected for the locations visual elements        410 (marker) and the shape can be filled or empty. The outline        thickness can also be adjusted. The default setting can be a        circle and can indicate spatial precision with size. For        example, more precise locations, such as addresses, are smaller        and have thicker line width, whereas a less precise location is        larger in diameter, but uses a thin line width.    -   The Location visual elements 410 are also sensitive to        interaction. The following interactions are possible:

-   Mouse-Left-Click:    -   Selects the location visual element 410 and highlights it, while        deselecting any previously selected location visual elements        410.

-   Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click    -   Adds the location visual element 410 to an existing selection        set.

-   Mouse-Left-Double-Click:    -   Opens a file specified in a Location data parameter if it        exists. The file will be opened in a system-specified default        application window based on its file type.

-   Mouse-Right-Click:    -   Displays an in-context popup menu with options to hide, delete        and set properties of the location visual element 410.

-   Mouseover Drilldown:    -   When the Mouse pointer is placed over the location indicator, a        text window showing information about the location visual        element 410 is displayed next to the pointer. When the mouse        pointer is moved away from the indicator, the text window        disappears.

-   Mouse-Left-Click-Hold-and-Drag:    -   Interactively repositions the location visual element 410 by        dragging it across the reference surface 404.        Non-Spatial Locations

Locations 22 have the ability to represent indeterminate position. Theseare referred to as non-spatial locations 22. Locations 22 tagged asnon-spatial can be displayed at the edge of the reference surface 404just outside of the spatial context of the spatial domain 400. Thesenon-spatial or virtual locations 22 can be always visible no matterwhere the user is currently zoomed in on the reference surface 404.Events and Timelines 422 that are associated with non-spatial Locations22 can be rendered the same way as Events with spatial Locations 22.

Further, it is recognized that spatial locations 22 can representactual, physical places, such that if the latitude/longitude is knownthe location 22 appears at that position on the map or if thelatitude/longitude is unknown the location 22 appears on the bottomcorner of the map (for example). Further, it is recognized thatnon-spatial locations 22 can represent places with no real physicallocation and can always appear off the right side of map (for example).For events 20, if the location 22 of the event 20 is known, the location22 appears at that position on the map. However, if the location 22 isunknown, the location 22 can appear halfway (for example) between thegeographical positions of the adjacent event locations 22 (e.g. part oftarget tracking)

Entity Representation

Entity visual elements 410 are represented by a glyph, or icon, and canbe positioned on the reference surface 404 or other area of the spatialdomain 400, based on associated Event data that specifies its positionat the current Moment of Interest 900 (see FIG. 9) (i.e. specific pointon the timeline 422 that intersects the reference surface 404). If thecurrent Moment of Interest 900 lies between 2 events in time thatspecify different positions, the Entity position will be interpolatedbetween the 2 positions. Alternatively, the Entity could be positionedat the most recent known location on the reference surface 404. TheEntity glyph is actually a group of the entity visual elements 410 (e.g.graphical objects, or layers) each of which expresses the content of theevent data object 20 in a different way. Each layer can be toggled andadjusted by the user on a per event basis, in groups or across all eventinstances. The entity visual elements 410 are such as but not limitedto:

1. Text Label

-   -   The Text label is a graphic object for displaying the name of        the Entity. This text always faces the viewer no matter how the        reference surface 404 is oriented. The text label incorporates a        de-cluttering function that separates it from other labels if        they overlap.

2. Indicator

-   -   The indicator is a point showing the interpolated or real        position of the Entity in the spatial context of the reference        surface 404. The indicator assumes the color specified as an        Entity color in the Entity data model.

3. Image Icon

-   -   An icon or image is displayed at the Entity location. This icon        may used to represent the identity of the Entity. The displayed        image can be user-specified or entered as part of a data file.        The Image Icon can have an outline border that assumes the color        specified as the Entity color in the Entity data model. The        Image Icon incorporates a de-cluttering function that separates        it from other Entity Image Icons if they overlap.

4. Past Trail

-   -   The Past Trail is the connection visual element 412, as a series        of connected lines that trace previous known positions of the        Entity over time, starting from the current Moment of Interest        900 and working backwards into past time of the timeline 422.        Previous positions are defined as Events where the Entity was        known to be located. The Past Trail can mark the path of the        Entity over time and space simultaneously.

5. Future Trail

-   -   The Future Trail is the connection visual element 412, as a        series of connected lines that trace future known positions of        the Entity over time, starting from the current Moment of        Interest 900 and working forwards into future time. Future        positions are defined as Events where the Entity is known to be        located. The Future Trail can mark the future path of the Entity        over time and space simultaneously.

The Entity representation is also sensitive to interaction. Thefollowing interactions are possible, such as but not limited to:

-   Mouse-Left-Click:    -   Selects the entity visual element 410 and highlights it and        deselects any previously selected entity visual element 410.-   Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click    -   Adds the entity visual element 410 to an existing selection set-   Mouse-Left-Double-Click:    -   Opens the file specified in an Entity data parameter if it        exists. The file will be opened in a system-specified default        application window based on its file type.-   Mouse-Right-Click:    -   Displays an in-context popup menu with options to hide, delete        and set properties of the entity visual element 410.-   Mouseover Drilldown:    -   When the Mouse pointer is placed over the indicator, a text        window showing information about the entity visual element 410        is displayed next to the pointer. When the mouse pointer is        moved away from the indicator, the text window disappears.        Temporal Domain Including Timelines

Referring to FIGS. 8 and 9, the temporal domain provides a commontemporal reference frame for the spatial domain 400, whereby the domains400, 402 are operatively coupled to one another to simultaneouslyreflect changes in interconnected spatial and temporal properties of thedata elements 14 and associations 16. Timelines 422 (otherwise known astime tracks) represent a distribution of the temporal domain 402 overthe spatial domain 400, and are a primary organizing element ofinformation in the visualization representation 18 that make it possibleto display events across time within the single spatial display on theVI 202 (see FIG. 1). Timelines 422 represent a stream of time through aparticular Location visual element 410 a positioned on the referencesurface 404 and can be represented as a literal line in space. Otheroptions for representing the timelines/time tracks 422 are such as butnot limited to curved geometrical shapes (e.g. spirals) including 2D and3D curves when combining two or more parameters in conjunction with thetemporal dimension. Each unique Location of interest (represented by thelocation visual element 410 a) has one Timeline 422 that passes throughit. Events (represented by event visual elements 410 b) that occur atthat Location are arranged along this timeline 422 according to theexact time or range of time at which the event occurred. In this waymultiple events (represented by respective event visual elements 410 b)can be arranged along the timeline 422 and the sequence made visuallyapparent. A single spatial view will have as many timelines 422 asnecessary to show every Event at every location within the currentspatial and temporal scope, as defined in the spatial 400 and temporal402 domains (see FIG. 4) selected by the user. In order to makecomparisons between events and sequences of event between locations, thetime range represented by multiple timelines 422 projecting through thereference surface 404 at different spatial locations is synchronized. Inother words the time scale is the same across all timelines 422 in thetime domain 402 of the visual representation 18. Therefore, it isrecognised that the timelines 422 are used in the visual representation18 to visually depict a graphical visualization of the data objects 14over time with respect to their spatial properties/attributes.

For example, in order to make comparisons between events 20 andsequences of events 20 between locations 410 of interest (see FIG. 4),the time range represented by the timelines 422 can be synchronized. Inother words, the time scale can be selected as the same for everytimeline 422 of the selected time range of the temporal domain 402 ofthe representation 18.

Representing Current, Past and Future

Three distinct strata of time are displayed by the timelines 422,namely;

1. The “moment of interest” 900 or browse time, as selected by the user,

2. a range 902 of past time preceding the browse time called “past”, and

3. a range 904 of time after the moment of interest 900, called “future”

On a 3D Timeline 422, the moment of focus 900 is the point at which thetimeline intersects the reference surface 404. An event that occurs atthe moment of focus 900 will appear to be placed on the referencesurface 404 (event representation is described above). Past and futuretime ranges 902, 904 extend on either side (above or below) of themoment of interest 900 along the timeline 422. Amount of time into thepast or future is proportional to the distance from the moment of focus900. The scale of time may be linear or logarithmic in either direction.The user may select to have the direction of future to be down and pastto be up or vice versa.

There are three basic variations of Spatial Timelines 422 that emphasizespatial and temporal qualities to varying extents. Each variation has aspecific orientation and implementation in terms of its visualconstruction and behavior in the visualization representation 18 (seeFIG. 1). The user may choose to enable any of the variations at any timeduring application runtime, as further described below.

3D Z-Axis Timelines

FIG. 10 shows how 3D Timelines 422 pass through reference surface 404locations 410 a. 3D timelines 422 are locked in orientation (angle) withrespect to the orientation of the reference surface 404 and are affectedby changes in perspective of the reference surface 404 about theviewpoint 420 (see FIG. 8). For example, the 3D Timelines 422 can beoriented normal to the reference surface 404 and exist within itscoordinate space. Within the 3D spatial domain 400, the referencesurface 404 is rendered in the X-Y plane and the timelines 422 runparallel to the Z-axis through locations 410 a on the reference surface404. Accordingly, the 3D Timelines 422 move with the reference surface404 as it changes in response to user navigation commands and viewpointchanges about the viewpoint 420, much like flag posts are attached tothe ground in real life. The 3D timelines 422 are subject to the sameperspective effects as other objects in the 3D graphical window of theVI 202 (see FIG. 1) displaying the visual representation 18. The 3DTimelines 422 can be rendered as thin cylindrical volumes and arerendered only between events 410 a with which it shares a location andthe location 410 a on the reference surface 404. The timeline 422 mayextend above the reference surface 404, below the reference surface 404,or both. If no events 410 b for its location 410 a are in view thetimeline 422 is not shown on the visualization representation 18.

3D Viewer Facing Timelines

Referring to FIG. 8, 3D Viewer-facing Timelines 422 are similar to 3DTimelines 422 except that they rotate about a moment of focus 425 (pointat which the viewing ray of the viewpoint 420 intersects the referencesurface 404) so that the 3D Viewer-facing Timeline 422 always remainperpendicular to viewer 423 from which the scene is rendered. 3DViewer-facing Timelines 422 are similar to 3D Timelines 422 except thatthey rotate about the moment of focus 425 so that they are alwaysparallel to a plane 424 normal to the viewing ray between the viewer 423and the moment of focus 425. The effect achieved is that the timelines422 are always rendered to face the viewer 423, so that the length ofthe timeline 422 is always maximized and consistent. This techniqueallows the temporal dimension of the temporal domain 402 to be read bythe viewer 423 indifferent to how the reference surface 404 many beoriented to the viewer 423. This technique is also generally referred toas “billboarding” because the information is always oriented towards theviewer 423. Using this technique the reference surface 404 can be viewedfrom any direction (including directly above) and the temporalinformation of the timeline 422 remains readable.

Linked TimeChart Timelines

Referring to FIG. 11, showing how an overlay time chart 430 is connectedto the reference surface 404 locations 410 a by timelines 422. Thetimelines 422 of the Linked TimeChart 430 are timelines 422 that connectthe 2D chart 430 (e.g. grid) in the temporal domain 402 to locations 410a marked in the 3D spatial domain 400. The timeline grid 430 is renderedin the visual representation 18 as an overlay in front of the 2D or 3Dreference surface 404. The timeline chart 430 can be a rectangularregion containing a regular or logarithmic time scale upon which eventrepresentations 410 b are laid out. The chart 430 is arranged so thatone dimension 432 is time and the other is location 434 based on theposition of the locations 410 a on the reference surface 404. As thereference surface 404 is navigated or manipulated the timelines 422 inthe chart 430 move to follow the new relative location 410 a positions.This linked location and temporal scrolling has the advantage that it iseasy to make temporal comparisons between events since time isrepresented in a flat chart 430 space. The position 410 b of the eventcan always be traced by following the timeline 422 down to the referencesurface 404 to the location 410 a.

Referring to FIGS. 11 and 12, the TimeChart 430 can be rendered in 2orientations, one vertical and one horizontal. In the vertical mode ofFIG. 11, the TimeChart 430 has the location dimension 434 shownhorizontally, the time dimension 432 vertically, and the timelines 422connect vertically to the reference surface 404. In the horizontal modeof FIG. 12, the TimeChart 430 has the location dimension 434 shownvertically, the time dimension 432 shown horizontally and the timelines422 connect to the reference surface 404 horizontally. In both cases theTimeChart 430 position in the visualization representation 18 can bemoved anywhere on the screen of the VI 202 (see FIG. 1), so that thechart 430 may be on either side of the reference surface 404 or in frontof the reference surface 404. In addition, the temporal directions ofpast 902 and future 904 can be swapped on either side of the focus 900.Interaction Interface Descriptions

Referring to FIGS. 3 and 13, several interactive controls 306 supportnavigation and analysis of information within the visualizationrepresentation 12, as monitored by the visualization manger 300 inconnection with user events 109. Examples of the controls 306 are suchas but not limited to a time slider 910, an instant of focus selector912, a past time range selector 914, and a future time selector 916. Itis recognized that these controls 306 can be represented on the VI 202(see FIG. 1) as visual based controls, text controls, and/or acombination thereof.

Time and Range Slider 901

The timeline slider 910 is a linear time scale that is visibleunderneath the visualization representation 18 (including the temporal402 and spatial 400 domains). The control 910 contains subcontrols/selectors that allow control of three independent temporalparameters: the Instant of Focus, the Past Range of Time and the FutureRange of Time.

Continuous animation of events 20 over time and geography can beprovided as the time slider 910 is moved forward and backwards in time.Example, if a vehicle moves from location A at t1 to location B at t2,the vehicle (object 23,24) is shown moving continuously across thespatial domain 400 (e.g. map). The timelines 422 can animate up and downat a selected frame rate in association with movement of the slider 910.

Instant of Focus

The instant of focus selector 912 is the primary temporal control. It isadjusted by dragging it left or right with the mouse pointer across thetime slider 910 to the desired position. As it is dragged, the Past andFuture ranges move with it. The instant of focus 900 (see FIG. 12) (alsoknown as the browse time) is the moment in time represented at thereference surface 404 in the spatial-temporal visualizationrepresentation 18. As the instant of focus selector 912 is moved by theuser forward or back in time along the slider 910, the visualizationrepresentation 18 displayed on the interface 202 (see FIG. 1) updatesthe various associated visual elements of the temporal 402 and spatial400 domains to reflect the new time settings. For example, placement ofEvent visual elements 410 animate along the timelines 422 and Entityvisual elements 410 move along the reference surface 404 interpolatingbetween known locations visual elements 410 (see FIGS. 6 and 7).Examples of movement are given with reference to FIGS. 14, 15, and 16below.

Past Time Range

The Past Time Range selector 914 sets the range of time before themoment of interest 900 (see FIG. 11) for which events will be shown. ThePast Time range is adjusted by dragging the selector 914 left and rightwith the mouse pointer. The range between the moment of interest 900 andthe Past time limit can be highlighted in red (or other colour codings)on the time slider 910. As the Past Time Range is adjusted, viewingparameters of the spatial-temporal visualization representation 18update to reflect the change in the time settings.

Future Time Range

The Future Time Range selector 914 sets the range of time after themoment of interest 900 for which events will be shown. The Future Timerange is adjusted by dragging the selector 916 left and right with themouse pointer. The range between the moment of interest 900 and theFuture time limit is highlighted in blue (or other colour codings) onthe time slider 910. As the Future Time Range is adjusted, viewingparameters of the spatial-temporal visualization representation 18update to reflect the change in the time settings.

The time range visible in the time scale of the time slider 910 can beexpanded or contracted to show a time span from centuries to seconds.Clicking and dragging on the time slider 910 anywhere except the threeselectors 912, 914, 916 will allow the entire time scale to slide totranslate in time to a point further in the future or past. Othercontrols 918 associated with the time slider 910 can be such as a “Fit”button 919 for automatically adjusting the time scale to fit the rangeof time covered by the currently active data set displayed in thevisualization representation 18. Controls 918 can include a Fit control919, a scale-expand-contract controls 920, a step control 923, and aplay control 922, which allow the user to expand or contract the timescale. A step control 918 increments the instant of focus 900 forward orback. The “playback” button 920 causes the instant of focus 900 toanimate forward by a user-adjustable rate. This “playback” causes thevisualization representation 18 as displayed to animate in sync with thetime slider 910.

Simultaneous Spatial and Temporal Navigation can be provided by the tool12 using, for example, interactions such as zoom-box selection and savedviews. In addition, simultaneous spatial and temporal zooming can beused to provide the user to quickly move to a context of interest. Inany view of the representation 18, the user may select a subset ofevents 20 and zoom to them in both time 402 and space 400 domains usinga Fit Time and a Fit Space functions. These functions can happensimultaneously by dragging a zoom-box on to the time chart 430 itself.The time range and the geographic extents of the selected events 20 canbe used to set the bounds of the new view of the representation 18,including selected domain 400,402 view formats.

Referring again to FIGS. 13 and 27, the Fit control 919 of the timerslider and other controls 306 can be further subdivided into separatefit time and fit geography/space functions as performed by a fit module700. For example, with a single click via the controls 306, for the fitto geography function the fit module 700 can instruct the visualizationmanager 300 to zoom in to user selected objects 20,21,22,23,24 (i.e.visual elements 410) and/or connection elements 412 (see FIG. 17) inboth/either space (FG) and/or time (FT), as displayed in a re-rendered“fit” version of the representation 18. For example, for fit togeography, after the user has selected places, targets and/or events(i.e. elements 410,412) from the representation 18, the fit module 700instructs the visualization manager 300 to reduce/expand the displayedmap of the representation 18 to only the geographic area that includesthose selected elements 410,412. If nothing is selected, the map isfitted to the entire data set (i.e. all geographic areas) included inthe representation 18. For example, for fit to time, after the user hasselected places, targets and/or events (i.e. elements 410,412) from therepresentation 18, the fit module 700 instructs the visualizationmanager 300 to reduce/expand the past portion of the timeline(s) 422 toencompass only the period that includes the selected visual elements410,412. Further, the fit module 700 can instruct the visualizationmanager 300 to adjust the display of the browse time slider as moved tothe end of the period containing the selected visual elements 410,412and the future portion of the timeline 422 can account for the sameproportion of the visible timeline 422 as it did before the timeline(s)422 were “time fitted”. If nothing is selected, the timeline is fittedto the entire data set (i.e. all temporal areas) included in therepresentation 18. Further, it is recognized, for both Fit to Geographyand Fit to Timeline, if only targets are selected, the fit module 700coordinates the display of the map/timeline to fit to the targets'entire set of events. Further for example, if a target is selected inaddition to events, only those events selected are used in the fitcalculation of the fit module 700.

Association Analysis Tools

Referring to FIGS. 1 and 3, an association analysis module 307 hasfunctions that have been developed that take advantage of theassociation-based connections between Events, Entities and Locations.These functions 307 are used to find groups of connected objects 14during analysis. The associations 16 connect these basic objects 20, 22,24 into complex groups 27 (see FIGS. 6 and 7) representing actualoccurrences. The functions are used to follow the associations 16 fromobject 14 to object 14 to reveal connections between objects 14 that arenot immediately apparent. Association analysis functions are especiallyuseful in analysis of large data sets where an efficient method to findand/or filter connected groups is desirable. For example, an Entity 24maybe be involved in events 20 in a dozen places/locations 22, and eachof those events 20 may involve other Entities 24. The associationanalysis function 307 can be used to display only those locations 22 onthe visualization representation 18 that the entity 24 has visited orentities 24 that have been contacted.

The analysis functions A,B,C,D provide the user with different types oflink analysis that display connections between 14 of interest, such asbut limited to:

1. Expanding Search A, e.g. a Link Analysis Tool

-   -   The expanding search function A of the module 307 allows the        user to start with a selected object(s) 14 and then        incrementally show objects 14 that are associated with it by        increasing degrees of separation. The user selects an object 14        or group of objects 14 of focus and clicks on the Expanding        search button 920 this causes everything in the visualization        representation 18 to disappear except the selected items. The        user then increments the search depth (e.g. via an appropriate        depth slider control) and objects 14 connected by the specified        depth are made visible the display. In this way, sets of        connected objects 14 are revealed as displayed using the visual        elements 410 and 412.    -   Accordingly, the function A of the module 307 displays all        objects 14 in the representation 18 that are connected to a        selected object 14, within the specified range of separation.        The range of separation of the function A can be selected by the        user using the I/O interface 108, using a links slider 730 in a        dialog window (see FIG. 31 a). For example, this link analysis        can be performed when a single place 22, target 24 or event 20        is first selected. An example operation of the depth slider is        as follows, when the function A is first selected via the I/O        interface 108, a dialog opens, and the links slider is initially        set to 0 and only the selected object 14 is displayed in the        representation 18. Using the slider (or entry field), when the        links slider is moved to 1, any object 14 directly linked (i.e.        1 degree of separation such as all elementary events 20) to the        initially selected object 14 appears on the representation 18 in        addition to the initially selected object 14. As the links        slider is positioned higher up the slider scale, additional        connected objects are added at each level to the representation        18, until all objects connected to the initially selected object        14 are displayed.

2. Connection Search B, e.g. a Join Analysis Tool

-   -   The Connection Search function B of the module 307 allows the        user to connect any pair of objects 14 by their web of        associations 26. The user selects any two objects 14 and clicks        on the Connection Search function B. The connection search        function B works by automatically scanning the extents of the        web of associations 26 starting from one of the initially        selected objects 14 of the pair. The search will continue until        the second object 14 is found as one of the connected objects 14        or until there are no more connected objects 14. If a path of        associated objects 14 between the target objects 14 exists, all        of the objects 14 along that path are displayed and the depth is        automatically displayed showing the minimum number of links        between the objects 14.    -   Accordingly, the Join Analysis function B looks for and displays        any specified connection path between two selected objects 14.        This join analysis is performed when two objects 14 are selected        from the representation 18. It is noted that if the two selected        objects 14 are not connected, no events 20 are displayed and the        connection level is set to zero on the display 202 (see FIG. 1).        If the paired objects 14 are connected, the shortest path        between them is automatically displayed, for example. It is        noted that the Join Analysis function B can be generalized for        three or more selected objects 14 and their connections. An        example operation of the Join Analysis function B is a selection        of the targets 24 Alan and Rome. When the dialog opens, the        number of links 732 (e.g. 4—which is user adjustable—see FIG. 31        b) required to make a connection between the two targets 24 is        displayed to the user, and only the objects 14 involved in that        connection (having 4 links) are visible on the representation        18.

3. A Chain Analysis Tool C

-   -   The Chain Analysis Tool C displays direct and/or indirect        connections between a selected target 24 and other targets 24.        For example, in a direct connection, a single event 20 connects        target A and target B (who are both on the terrain 400). In an        indirect connection, some number of events 20 (chain) connect A        and B, via a target C (who is located off the terrain 400 for        example). This analysis C can be performed with a single initial        target 24 selected. For example, the tool C can be associated        with a chaining slider 736—see FIG. 31 c (accessed via the I/O        interface 108) with the selections of such as but not limited to        direct, indirect, and both. For example, the target TOM is first        selected on the representation 18 and then when the target        chaining slider is set to Direct, the targets ALAN and PARENTS        are displayed, along with the events that cause TOM to be        directly connected to them. In the case where TOM does not have        any indirect target 24 connections, so moving the slider to Both        and to Indirect does not change the view as generated on the        representation 18 for the Direct chaining slider setting.

4. A Move Analysis Tool D

-   -   This tool D finds, for a single target 24, all sets of        consecutive events 20, that are located at different places 22        that happened within the specific time range of the temporal        domain 402. For example, this analysis of tool D may be        performed with a single target 24 selected from the        representation 18. In example operation of the tool D, the        initial target 24 is selected, when a slider 736 opens, the time        range slider 736 is set to one Year and quite a few connected        events 20 may be displayed on the representation 18, which are        connected to the initially selected target 24. When the slider        736 selection is changed to the unit type of one Week, the        number of events 20 displayed will drop accordingly. Similarly,        as the time range slider 736 is positioned higher, the number of        events 20 are added to the representation 18 as the time range        increases.

It is recognized that the functions of the module 307 can be used toimplement filtering via such as but not limited to criteria matching,algorithmic methods and/or manual selection of objects 14 andassociations 16 using the analytical properties of the tool 12. Thisfiltering can be used to highlight/hide/show (exclusively) selectedobjects 14 and associations 16 as represented on the visualrepresentation 18. The functions are used to create a group (subset) ofthe objects 14 and associations 16 as desired by the user through thespecified criteria matching, algorithmic methods and/or manualselection. Further, it is recognized that the selected group of objects14 and associations 16 could be assigned a specific name, which isstored in the table 122.

Operation of Visual Tool to Generate Visualization Representation

Referring to FIG. 14, example operation 1400 shows communications 1402and movement events 1404 (connection visual elements 412—see FIGS. 6 and7) between Entities “X” and “Y” over time on the visualizationrepresentation 18. This FIG. 14 shows a static view of Entity X makingthree phone call communications 1402 to Entity Y from 3 differentlocations 410 a at three different times. Further, the movement events1404 are shown on the visualization representation 18 indicating thatthe entity X was at three different locations 410 a (location A,B,C),which each have associated timelines 422. The timelines 422 indicate bythe relative distance (between the elements 410 b and 410 a) of theevents (E1,E2,E3) from the instant of focus 900 of the reference surface404 that these communications 1404 occurred at different times in thetime dimension 432 of the temporal domain 402. Arrows on thecommunications 1402 indicate the direction of the communications 1402,i.e. from entity X to entity Y. Entity Y is shown as remaining at onelocation 410 a (D) and receiving the communications 1402 at thedifferent times on the same timeline 422.

Referring to FIG. 15, example operation 1500 for shows Events 140 boccurring within a process diagram space domain 400 over the timedimension 432 on the reference surface 404. The spatial domain 400represents nodes 1502 of a process. This FIG. 14 shows how a flowchartor other graphic process can be used as a spatial context for analysis.In this case, the object (entity) X has been tracked through theproduction process to the final stage, such that the movements 1504represent spatial connection elements 412 (see FIGS. 6 and 7).

Referring to FIGS. 3 and 19, operation 800 of the tool 12 begins by themanager 300 assembling 802 the group of objects 14 from the tables 122via the data manager 114. The selected objects 14 are combined 804 viathe associations 16, including assigning the connection visual element412 (see FIGS. 6 and 7) for the visual representation 18 betweenselected paired visual elements 410 corresponding to the selectedcorrespondingly paired data elements 14 of the group. The connectionvisual element 412 represents a distributed association 16 in at leastone of the domains 400, 402 between the two or more paired visualelements 410. For example, the connection element 412 can representmovement of the entity object 24 between locations 22 of interest on thereference surface 404, communications (money transfer, telephone call,email, etc. . . . ) between entities 24 different locations 22 on thereference surface 404 or between entities 24 at the same location 22, orrelationships (e.g. personal, organizational) between entities 24 at thesame or different locations 22.

Next, the manager 300 uses the visualization components 308 (e.g.sprites) to generate 806 the spatial domain 400 of the visualrepresentation 18 to couple the visual elements 410 and 412 in thespatial reference frame at various respective locations 22 of interestof the reference surface 404. The manager 300 then uses the appropriatevisualization components 308 to generate 808 the temporal domain 402 inthe visual representation 18 to include various timelines 422 associatedwith each of the locations 22 of interest, such that the timelines 422all follow the common temporal reference frame. The manager 112 thentakes the input of all visual elements 410, 412 from the components 308and renders them 810 to the display of the user interface 202. Themanager 112 is also responsible for receiving 812 feedback from the uservia user events 109 as described above and then coordinating 814 withthe manager 300 and components 308 to change existing and/or create (viasteps 806, 808) new visual elements 410, 412 to correspond to the userevents 109. The modified/new visual elements 410, 412 are then renderedto the display at step 810.

Referring to FIG. 16, an example operation 1600 shows animating entity Xmovement between events (Event 1 and Event 2) during time slider 901interactions via the selector 912. First, the Entity X is observed atLocation A at time t. As the slider selector 912 is moved to the right,at time t+1 the Entity X is shown moving between known locations (Event1and Event2). It should be noted that the focus 900 of the referencesurface 404 changes such that the events 1 and 2 move along theirrespective timelines 422, such that Event 1 moves from the future intothe past of the temporal domain 402 (from above to below the referencesurface 404). The length of the timeline 422 for Event 2 (between theEvent 2 and the location B on the reference surface 404 decreasesaccordingly. As the slider selector 912 is moved further to the right,at time t+2, Entity X is rendered at Event2 (Location B). It should benoted that the Event 1 has moved along its respective timeline 422further into the past of the temporal domain 402, and event 2 has movedaccordingly from the future into the past of the temporal domain 402(from above to below the reference surface 404), since therepresentation of the events 1 and 2 are linked in the temporal domain402. Likewise, the entity X is linked spatially in the spatial domain400 between event 1 at location A and event 2 at location B. It is alsonoted that the Time Slider selector 912 could be dragged along the timeslider 910 by the user to replay the sequence of events from time t tot+2, or from t+2 to t, as desired.

Referring to FIG. 27, a further feature of the tool 12 is a targettracing module 722, which takes user input from the I/O interface 108for tracing of a selected target/entity 24 through associated events 20.For example, the user of the tool 12 selects one of the events 20 fromthe representation 18 associated with one or more entities/target 24,whereby the module 722 provides for a selection icon to be displayedadjacent to the selected event 20 on the representation 18. Using theinterface 108 (e.g. up/down arrows), the user can navigate therepresentation 18 by scrolling back and forward (in terms of time and/orgeography) through the events 20 associated with that target 24, i.e.the display of the representation 18 adapts as the user scrolls throughthe time domain 402, as described already above. For example, thedisplay of the representation 18 moves between consecutive events 20associated with the target 24. In an example implementation of the I/Ointerface 108, the Page Up key moves the selection icon upwards (back intime) and the Page Down key moves the selection icon downwards (forwardin time), such that after selection of a single event 20 with anassociated target 24, the Page Up keyboard key would move the selectionicon to the next event 20 (back in time) on the associated target'strail while selecting the Page Down key would return the selection iconto the first event 20 selected. The module 722 coordinates placement ofthe selection icon at consecutive events 20 connected with theassociated target 24 while skipping over those events 20 (whilescrolling) not connected with the associated target 24.

Referring to FIG. 17, the visual representation 18 shows connectionvisual elements 412 between visual elements 410 situated on selectedvarious timelines 422. The timelines 422 are coupled to variouslocations 22 of interest on the geographical reference frame 404. Inthis case, the elements 412 represent geographical movement betweenvarious locations 22 by entity 24, such that all travel happened at sometime in the future with respect to the instant of focus represented bythe reference plane 404.

Referring to FIG. 18, the spatial domain 400 is shown as a geographicalrelief map. The timechart 430 is superimposed over the spatial domain ofthe visual representation 18, and shows a time period spanning fromDecember 3^(rd) to January 1^(st) for various events 20 and entities 24situated along various timelines 422 coupled to selected locations 22 ofinterest. It is noted that in this case the user can use the presentedvisual representation to coordinate the assignment of various connectionelements 412 to the visual elements 410 (see FIG. 6) of the objects 20,22, 24 via the user interface 202 (see FIG. 1), based on analysis of thedisplayed visual representation 18 content. A time selection 950 isJanuary 30, such that events 20 and entities 24 within the selection boxcan be further analysed. It is recognised that the time selection 950could be used to represent the instant of focus 900 (see FIG. 9).

Aggregation Module 600

Referring to FIG. 3, an Aggregation Module 600 is for, such as but notlimited to, summarizing or aggregating the data objects 14, providingthe summarized or aggregated data objects 14 to the VisualizationManager 300 which processes the translation from data objects 14 andgroup of data elements 27 to the visual representation 18, and providingthe creation of summary charts 200 (see FIG. 26) for displayinginformation related to summarised/aggregated data objects 14 as thevisual representation 18 on the display 108.

Referring to FIGS. 3 and 22, the spatial inter-connectedness ofinformation over time and geography within a single, highly interactive3-D view of the representation 18 is beneficial to data analysis (of thetables 122). However, when the number of data objects 14 increases,techniques for aggregation become more important. Many individuallocations 22 and events 20 can be combined into a respective summary oraggregated output 603. Such outputs 603 of a plurality of individualevents 20 and locations 22 (for example) can help make trends in timeand space domains 400,402 more visible and comparable to the user of thetool 12. Several techniques can be implemented to support aggregation ofdata objects 14 such as but not limited to techniques of hierarchy oflocations, user defined geo-relations, and automatic LOD levelselection, as further described below. The tool 12 combines the spatialand temporal domains 400, 402 on the display 108 for analysis of complexpast and future events within a selected spatial (e.g. geographic)context.

Referring to FIG. 22, the Aggregation Module 600 has an AggregationManager 601 that communicates with the Visualization Manager 300 forreceiving aggregation parameters used to formulate the output 603 as apattern aggregate 62 (see FIGS. 23, 24). The parameters can be eitherautomatic (e.g. tool pre-definitions) manual (entered via events 109) ora combination thereof. The manager 601 accesses all possible dataobjects 14 through the Data Manager 114 (related to the aggregationparameters—e.g. time and/or spatial ranges and/or object 14types/combinations) from the tables 122, and then applies aggregationtools or filters 602 for generating the output 603. The VisualizationManager 300 receives the output 603 from the Aggregation Manager 601,based on the user events 109 and/or operation of the Time Slider andother Controls 306 by the user for providing the aggregation parameters.As described above, once the output 603 is requested by theVisualization Manager 114, the Aggregation Manager 601 communicates withthe Data Manager 114 access all possible data objects 14 for satisfyingthe most general of the aggregation parameters and then applies thefilters 602 to generate the output 603. It is recognised however, thatthe filters 602 could be used by the manager 601 to access only thosedata objects 14 from the tables 122 that satisfy the aggregationparameters, and then copy those selected data objects 14 from the tables122 for storing/mapping as the output 603.

Accordingly, the Aggregation Manager 601 can make available the dataelements 14 to the Filters 602. The filters 602 act to organize andaggregate (such as but not limited to selection of data objects 14 fromthe global set of data in the tables 122 according to rules/selectioncriteria associated with the aggregation parameters) the data objects 14according the instructions provided by the Aggregation Manager 601. Forexample, the Aggregation Manager 601 could request that the Filters 602summarize all data objects 14 with location data 22 corresponding toParis to compose the pattern aggregate 62. Or, in another example, theAggregation Manager 601 could request that the Filters 602 summarize alldata objects 14 with event data 20 corresponding to Wednesdays tocompose the pattern aggregate 62. Once the data objects 14 are selectedby the Filters 602, the aggregated data is summarised as the output 603.The Aggregation Manager 601 then communicates the output 603 to theVisualization Manager 300, which processes the translation from theselected data objects 14 (of the aggregated output 603) for rendering asthe visual representation 18 to include these to compose the patternaggregates 62. It is recognised that the content of the representation18 is modified to display the output 603 to the user of the tool 12,according to the aggregation parameters.

Further, the Aggregation Manager 601 provides the aggregated dataobjects 14 of the output 603 to a Chart Manager 604. The Chart Manager604 compiles the data in accordance with the commands it receives fromthe Aggregation Manager 601 and then provides the formatted data to aChart Output 605. The Chart Output 605 provides for storage of theaggregated data in a Chart section 606 of the display (see FIG. 25).Data from the Chart Output 605 can then be sent directly to theVisualization Renderer 112 or to the visualisation manager 300 forinclusion in the visual representation 18, as further described below.

Referring to FIG. 23, an example aggregation of data objects 14 as thepattern aggregate 62 by the Aggregation Module 601 is shown. The eventdata 20 (for example) is aggregated according to spatial proximity(threshold) of the data objects 14 with respect to a common point (e.g.particular location 410 or other newly specified point of the spatialdomain 400), difference threshold between two adjacent locations 410, orother spatial criteria as desired. For example, as depicted in FIG. 23a, the three data objects 20 at three locations 410 are aggregated totwo objects 20 at one location 410 and one object at another location410 (e.g. combination of two locations 410) as a user-defined field 202of view is reduced in FIG. 23 b, and ultimately to one location 410 withall three objects 20 in FIG. 23 c. It is recognised in this example ofaggregated output 603 that timelines 422 of the locations 410 arecombined as dictated by the aggregation of locations 410.

For example, the user may desire to view an aggregate of data objects 14related within a set distance of a fixed location, e.g., aggregate ofevents 20 occurring within 50 km of the Golden Gate Bridge. Toaccomplish this, the user inputs their desire to aggregate the dataaccording to spatial proximity, by use of the controls 306, indicatingthe specific aggregation parameters. The Visualization Manager 300communicates these aggregation parameters to the Aggregation Module 600,in order for filtering of the data content of the representation 18shown on the display 108. The Aggregation Module 600 uses the Filters602 to filter the selected data from the tables 122 based on theproximity comparison between the locations 410. In another example, ahierarchy of locations can be implemented by reference to theassociation data 26 which can be used to define parent-childrelationships between data objects 14 related to specific locationswithin the representation 18. The parent-child relationships can be usedto define superior and subordinate locations that determine the level ofaggregation of the output 603.

Referring to FIG. 24, an example aggregation of data objects 14 tocompose the pattern aggregate 62 by the Aggregation Module 601 is shown.The data 14 is aggregated according to defined spatial boundaries 204.To accomplish this, the user inputs their desire to aggregate the data14 according to specific spatial boundaries 204, by use of the controls306, indicating the specific aggregation parameters of the filtering602. For example, a user may wish to aggregate all event 20 objectslocated within the city limits of Toronto. The Visualization Manager 300then requests to the Aggregation Module 600 to filter the data objects14 of the current representation according to the aggregationparameters. The Aggregation Module 600 provides implements or otherwiseapplies the filters 602 to filter the data based on a comparison betweenthe location data objects 14 and the city limits of Toronto, forgenerating the aggregated output 603 as the pattern aggregate 62. InFIG. 24 a, within the spatial domain 205 the user has specified tworegions of interest 204, each containing two locations 410 withassociated data objects 14. In FIG. 24 b, once filtering has beenapplied, the locations 410 of each region 204 have been combined suchthat now two locations 410 are shown with each having the aggregatedresult (output 603) of two data objects 14 respectively. In FIG. 24 c,the user has defined the region of interest to be the entire domain 205,thereby resulting in the displayed output 603 of one location 410 withthree aggregated data objects 14 (as compared to FIG. 24 a). It is notedthat the positioning of the aggregated location 410 is at the center ofthe regions of interest 204, however other positioning can be used suchas but not limited to spatial averaging of two or more locations 410 orplacing aggregated object data 14 at one of the retained originallocations 410, or other positioning techniques as desired.

In addition to the examples in illustrated in FIGS. 21 and 22, theaggregation of the data objects can be accomplished automatically basedon the geographic view scale provided in the visual representations.Aggregation can be based on level of detail (LOD) used in mappinggeographical features at various scales. On a 1:25,000 map, for example,individual buildings may be shown, but a 1:500,000 map may show just apoint for an entire city. The aggregation module 600 can supportautomatic LOD aggregation of objects 14 based on hierarchy, scale andgeographic region, which can be supplied as aggregation parameters aspredefined operation of the controls 306 and/or specific manualcommands/criteria via user input events 109. The module 600 can alsointeract with the user of the tool 12 (via events 109) to adjust LODbehaviour to suit the particular analytical task at hand.

Referring to FIG. 27 and FIG. 28, the aggregation module 600 can alsohave a place aggregation module 702 for assigning visual elements410,412 (e.g. events 20) of several places/locations 22 to one commonaggregation location 704, for the purpose of analyzing data for anentire area (e.g. a convoy route or a county). It is recognised that theplace aggregation function can be turned on and off for each aggregationlocation 704, so that the user of the tool 12 can analyze data with andwithout the aggregation(s) active. For example, the user creates theaggregation location 704 in a selected location of the spatial domain400 of the representation 18. The user then gives the createdaggregation location 704 a label 706 (e.g. North America). The user thenselects a plurality of locations 22 from the representation, eitherindividually or as a group using a drawing tool 707 to draw around alldesired locations 22 within a user defined region 708. Once selected,the user can drag or toggle the selected regions 708 and individuallocations 22 to be included in the created aggregation location 704 bythe aggregation module 702. The aggregation module 702 could instructthe visualization manager 300 to refresh the display of therepresentation 18 to display all selected locations 22 and relatedvisual elements 410,412 in the created aggregation location 704. It isrecognised that the aggregation module 702 could be used to configurethe created aggregation location 704 to display other selected objecttypes (e.g. entities 24) as a displayed group. In the case of selectedentities 24, the created aggregation location 704 could be labelled theselected entities' name and all visual elements 410,412 associated withthe selected entity (or entities) would be displayed in the createdaggregation location 704 by the aggregation module 702. It is recognisedthat the above-described same aggregation operation could be done forselected event 20 types, as desired.

Referring to FIG. 25, an example of a spatial and temporal visualrepresentation 18 with summary chart 200 depicting event data 20 isshown. For example, a user may wish to see the quantitative informationrelating to a specific event object. The user would request the creationof the chart 200 using the controls 306, which would submit the requestto the Visualization Manager 300. The Visualization Manager 300 wouldcommunicate with the Aggregation Module 600 and instruct the creation ofthe chart 200 depicting all of the quantitative information associatedwith the data objects 14 associated with the specific event object 20,and represent that on the display 108 (see FIG. 2) as content of therepresentation 18. The Aggregation Module 600 would communicate with theChart Manager 604, which would list the relevant data and provide onlythe relevant information to the Chart Output 605. The Chart Output 605provides a copy of the relevant data for storage in the Chart ComparisonModule, and the data output is communicated from the Chart Output 605 tothe Visualization Renderer 112 before being included in the visualrepresentation 18. The output data stored in the Chart Comparisonsection 606 can be used to compare to newly created charts 200 whenrequested from the user. The comparison of data occurs by selectingparticular charts 200 from the chart section 606 for application as theoutput 603 to the Visual Representation 18.

The charts 200 rendered by the Chart Manager 604 can be created in anumber of ways. For example, all the data objects 14 from the DataManager 114 can be provided in the chart 200. Or, the Chart Manager 604can filter the data so that only the data objects 14 related to aspecific temporal range will appear in the chart 200 provided to theVisual Representation 18. Or, the Chart Manager 604 can filter the dataso that only the data objects 14 related to a specific spatial andtemporal range will appear in the chart 200 provided to the VisualRepresentation 18.

Referring to FIG. 30, a further embodiment of event aggregation charts200 calculates and displays (both visually and numerically) the countobjects by various classifications 726. When charts 200 are displayed onthe map (e.g. on-map chart), one chart 200 is created for each place 22that is associated with relevant events 20. Additional options becomeavailable by clicking on the colored chart bars 728 (e.g. Hide selectedobjects, Hide target). By default, the chart manager 604 (see FIG. 22)can assign colors to chart bars 728 randomly, except for example whenthey are for targets 24, in which case the chart manager 604 usesexisting target 24 colors, for convenience. It is noted that a Chartscale slider 730 can be used to to increase or decrease the scale ofon-map charts 200, e.g. slide right or left respectively. The chartmanager 604 can generate the charts 200 based on user selected options724, such as but not limited to:

1) Show Charts on Map—presents a visual display on the map, one chart200 for each place 22 that has relevant events 20;

2) Chart Events in Time Range Only—includes only events 20 that happenedduring the currently selected time range;

3) Exclude Hidden Events—excludes events 20 that are not currentlyvisible on the display (occur within current time range, but arehidden);

4) Color by Event—when this option is turned on, event 20 color is usedfor any bar 728 that contains only events 20 of that one color. When abar 728 contains events 20 of more than one color, it is displayed gray;

5) Sort by Value—when turned on, results are displayed in the Charts 200panel, sorted by their value, rather than alphabetically; and

6) Show Advanced Options—gives access to additional statisticalcalculations.

In a further example of the aggregation module 601, user-definedlocation boundaries 204 can provide for aggregation of data 14 across anarbitrary region. Referring to FIG. 26, to compare a summary of eventsalong two separate routes 210 and 212, aggregation output 603 of thedata 14 associated with each route 210,212 would be created by drawingan outline boundary 204 around each route 210,212 and then assigning theboundaries 204 to the respective locations 410 contained therein, asdepicted in FIG. 26 a. By the user adjusting the aggregation level inthe Filters 602 through specification of the aggregation parameters ofthe boundaries 204 and associated locations 410, the data 14 is theaggregated as output 603 (see FIG. 26 b) within the outline regions intothe newly created locations 410, with the optional display of text 214providing analysis details for those new aggregated locations 410. Forexample, the text 214 could summarise that the number of bad events 20(e.g. bombings) is greater for route 210 than route 212 and thereforeroute 212 would be the route of choice based on the aggregated output603 displayed on the representation 18.

It will be appreciated that variations of some elements are possible toadapt the invention for specific conditions or functions. The conceptsof the present invention can be further extended to a variety of otherapplications that are clearly within the scope of this invention.

For example, one application of the tool 12 is in criminal analysis bythe “information producer”. An investigator, such as a police officer,could use the tool 12 to review an interactive log of events 20 gatheredduring the course of long-term investigations. Existing reports andquery results can be combined with user input data 109, assertions andhypotheses, for example using the annotations 21. The investigator canreplay events 20 and understand relationships between multiple suspects,movements and the events 20. Patterns of travel, communications andother types of events 20 can be analysed through viewing of therepresentation 18 of the data in the tables 122 to reveal such as butnot limited to repetition, regularity, and bursts or pauses in activity.

Subjective evaluations and operator trials with four subject matterexperts have been conducted using the tool 12. These initial evaluationsof the tool 12 were run against databases of simulated battlefieldevents and analyst training scenarios, with many hundreds of events 20.These informal evaluations show that the following types of informationcan be revealed and summarised. What significant events happened in thisarea in the last X days? Who was involved? What is the history of thisperson? How are they connected with other people? Where are the activityhot spots? Has this type of event occurred here or elsewhere in the lastY period of time?

With respect to potential applications and the utility of the tool 12,encouraging and positive remarks were provided by military subjectmatter experts in stability and support operations. A number of thoseremarks are provided here. Preparation for patrolling involvedresearching issues including who, where and what. The history of localbelligerent commanders and incidents. Tracking and being aware ofhistory, for example, a ceasefire was organized around a religiouscalendar event. The event presented an opportunity and knowing about theevent made it possible. In one campaign, the head of civil affairs hadbeen there twenty months and had detailed appreciation of the historyand relationships. Keeping track of trends. What happened here? Whatkeeps happening here? There are patterns. Belligerents keep trying thesame thing with new rotations [a rotation is typically six to twelvemonths tour of duty]. When the attack came, it did come from the areawhere many previous earlier attacks had also originated. The discoveryof emergent trends . . . persistent patterns . . . sooner rather thanlater could be useful. For example, the XXX Colonel that tends to showup in an area the day before something happens. For every rotation avaluable knowledge base can be created, and for every rotation, thisknowledge base can be retained using the tool 12 to make the knowledgebase a valuable historical record. The historical record can includeevents, factions, populations, culture, etc.

Referring to FIG. 27, the tool 12 could also have a report generationmodule 720 that saves a JPG format screenshot (or other picture format),with a title and description (optional—for example entered by the user)included in the screenshot image, of the visual representation 18displayed on the visual interface 202 (see FIG. 1). For example, thescreenshot image could include all displayed visual elements 410,412,including any annotations 21 or other user generated analysis related tothe displayed visual representation 18, as selected or otherwisespecified by the user. A default mode could be all currently displayedinformation is captured by the report generation module 720 and saved inthe screenshot image, along with the identifying label (e.g. titleand/or description as noted above) incorporated as part of thescreenshot image (e.g. superimposed on the lower right-hand corner ofthe image). Otherwise the user could select (e.g. from a menu) whichsubset of the displayed visual elements 410,412 (on acategory/individual basis) is for inclusion by the module 720 in thescreenshot image, whereby all non-selected visual elements 410,412 wouldnot be included in the saved screenshot image. The screenshot imagewould then be given to the data manager 114 (see FIG. 3) for storing inthe database 122. For further information detail of the visualrepresentation 18 not captured in the screenshot image, a filename (orother link such as a URL) to the non-displayed information could also besuperimposed on the screenshot image, as desired. Accordingly, the savedscreenshot image can be subsequently retrieved and used as a quickvisual reference for more detailed underlying analysis linked to thescreenshot image. Further, the link to the associated detailed analysiscould be represented on the subsequently displayed screenshot image as ahyperlink to the associated detailed analysis, as desired.

Visual Representation 18

Referring again to FIGS. 5, 6 and 7, shown are example visualrepresentations 18 of events over time and space in an x, y, t space, asproduced by the visualization tool 12. For example, in order to showthat a particular entity 24 was present at a location 22 at a certaintime, the entity 24 is paired with the event 20 which is in turn,attached to the location 22 present in the spatial domain 400. In allthree Figures, there exists a temporal domain (shown as the days in themonth in FIG. 5) 402, a spatial domain (showing the geographicallocations) 400 and connectivity elements 412. Thus, the visualizationtool 12 described above provides a visual analysis of entity 24activities, movements, and relationships as they change over time. Theoutput of the visualization tool 12 is the visual representation 18, asseen in FIG. 5 of the data objects 14 and associations 16 in atemporal-spatial display to show interconnecting stream of events 20 asthey change over the range of time associated with the spatial domain400. It is also recognized that stories 19 can be generated from datathat represents diagrammatic domains 401 as well as data that representsgeospatial domains 400, in view of interactions with the temporal domain402, as desired. Although this analysis and tracking of events 20 in thetime domain 402 and domain 400, 401 is useful in understanding certainbehaviours, including relationships and patterns of the entities 24 overtime, it is advantageous to provide visualization representations 18that depict the events, characters and locations in a “story” format.The story 19 (see FIG. 32) would conceptualize the raw data provided bythe data objects 14 (and/or associations 16) into a visual summary ofthe events 20 and entities 24 (for example) and will facilitate ananalyst to conceptualize the sequence (e.g. story elements 17) of eventsand possibly an expected result, as further described below.

Stories 19

Referring to FIGS. 1 and 32, a story 19 (also referred to as a storyframework) is an abstraction for use by analysts to conceptualizeconnected data (e.g. data objects 14 and associations 16) as part of theanalytical process, which offers a context for a connected collection ofthe data. Stories 19 are logical compositions of individual events 20,characters 24, locations 22 and sequences of these, for example. Thetool 12 supports the display of this story 19 type of information,including story elements 17 identified and labeled as such in order toconstruct the story 19. The story elements 17 are used as containers forthe story related evidence they describe, such that the visual form ofthe story elements 17 can be defined by their contents. Accordingly, thestory elements 17 can include a plurality of detailed informationaccessible to the user (e.g. though a mouse-over click-on or other userevent with respect to the selected story element 17), which is notimmediately apparent by viewing the associated semantic representation56 on the visual interface 202. For example, clicking on the semanticrepresentations 56 in FIG. 37 b would make available to the user theunderlying detail of the data subset 15 (see FIG. 37 a) associated withthe semantic representations 56. This underlying detail could replacethe semantic representation(s) 56 in the displayed story, could bedisplayed as a layer over the story, or could be displayed in a separatewindow or other version of the story, for example. The tool 12 is usedto construct the story from raw data collections in memory 102,including aggregation/clustering, pattern recognition, association ofsemantic context to represent the phase of story building, andassociation of the recognized story elements 17 as hyperlinks with astory text as written description of the story 19 used for storytelling.

Referring now to FIG. 33, shown are a plurality of semanticrepresentations 56 that describe the events 20 within the figure. Forexample, a telephone icon is used as a visual element 410 to showtelephone calls made between two parties or a money pouch symbol 56 toshow the transfer of money. Note that FIG. 33 also shows several patternaggregations shown as elements 66, 67 and 68. As illustrated in thisfigure, the display of pattern aggregates can be adjusted to representamount of raw data objects 14 replaced. The pattern aggregation 66 has arelatively thicker connection element 412 than the pattern aggregate 67and the pattern aggregate 68. In this example, the pattern aggregate 66has been used to replace 20 data objects (i.e. 17 phone calls made overtime involving 3 entities) while the pattern aggregate 67 replaces 10data objects and the pattern aggregate 68 replaces 2 data objects. Thus,the pattern aggregates 66, 67, and 68 visually depict the amount ofaggregation performed by the aggregation module 600, with or without theinteraction of the pattern module 60 in identifying the patterns 61 (seeFIG. 36).

From an analytical perspective, the story 19 is a logical, connectedcollection of characters 24, sequences of events 20 and relationshipsbetween characters, things and places over time. For example, referringto FIG. 33, shown is a visual representation 18 of the story 19generated from a story generation module 50 of FIG. 32. The story 19shows connecting visual elements 412 linking the sequence of events 20involving entities 24 in the temporal-spatial domains 402, 400.

For example, the stories 19 with coupling to the temporal and spatialdomains 402, 400, 401 could be used to understand problems such as, butnot limited to: generating of hypotheses and new possibilities, newlines of inquiry based on all available the data observations, includinglinks in time and geography/diagrams; putting all the facts together tosee how they relate to hypotheses, trajectories of facts over time tofacilitate telling of the story 19; constructing patterns in activitiesto reveal hidden information in the data when the whole puzzle is notself evident; identifying an easy pattern, for example, using the sameorganizations, the same timing, the same people; identifying a difficultpattern using different names, organizations, methods, dates; guidingthe organization of observations into meaningful structures and patternsthrough coherence and narrative principles; forming plots of dominantconcepts or leading ideas that the analyst use to postulate patterns ofrelationships among the data; and recognizing threads in a group ofpeople, or technologies, etc and then seeing other threads twistingthrough the situation. It is recognized that a hypothesis is anassertion while an elaborate hypothesis is a story.

Story 19 Interactions

Using an analytical tool 12 as a model, gesture-based interactions canbe used to enable story building, evidence marshaling, annotation, andpresentation. These interactions occur within the space-time environment402, 400, 401. Anticipated interactions are such as but not limited to:

-   -   Creation of a story fragments/elements 17 from nothing or from a        piece of evidence (as provided by the data objects 14);    -   Attaching and detaching evidence to story element structures        (i.e. the story 19);    -   Specify whether evidence supports or refutes the story 19;    -   Attaching elements 17 together;    -   Identifying “threads” in the story    -   Foreground/background/hidden modes for emphasis and focus of        story elements 17;    -   Perform pattern search within a constrained area of the source        data (e.g. data set in memory 102);    -   Creating annotations;    -   Removing junk; and    -   Automatic focus, navigation and animation controls of the story        19 once generated.

In addition, the tool 12 provides for the analyst to organize evidenceaccording to the story framework (series of connected story elements17). For example, the story framework (e.g. story 19) may allow analyststo sort or compare characters and events against templates for certaintype of threats.

Configuration of Tool 12 for Story 19 Generation

Referring to FIG. 32, shown is a system 113 for generating a visualrepresentation 18 of a series of data objects 14 including events 20,entities 24 and location 22. The events 20 and entities 24 are linked toeach other as defined by the associations data 16. The visualizationtool 12 processes the data objects 14, the associations data 16 receivedfrom a data manager 114. The data module 114, as provided by either auser or a database (e.g. memory 102), comprises data objects 14,associations data 16 defining the association between the data objects14 and pattern data 58 predefining the patterns (e.g. pattern templates59 used by the pattern module 60) between data objects 14 and/orassociations 16. In turn, the visualization tool 12 organizes somecombination of related data objects 14 in the context of spatial 400 andtemporal 402 domains, which in turn is subsequently identified as aspecific pattern 60 (e.g. compared to the raw data objects 14) and isincorporated into a story 19. Accordingly, the stories 19 or fragmentsof the stories 19 are then displayed as a visual representation 18 tothe user on the visual interface 202.

Story Generation Module 50

The story generation module 50 can be referred to as a workflow enginefor coordinating the generation of the story 19 through the connectionof a plurality of story elements 17 assigned to subsets of the dataobjects 14 and/or associations 16. The story generation module 50 usesqueries, pattern matching, and/or aggregation techniques to drive story19 development until a suitable story 19 is generated that representsthe data to which the story elements 17 are assigned. Ultimately, theoutput of the story generation module 50 is an assimilation of evidenceinto a series of connected data groups (e.g. story elements 17) withsemantic relevance to the story 19 as supported by the raw data from thememory 102. The story generation module 50 cooperates with theaggregation module 600 and the pattern module 60 to identify subsets 15of the data (see FIG. 37 a) and the semantic representation module 57 toattach semantic representations 56 (see FIG. 37 b) to the identifiedsubsets 15 in order to generate the story elements 17. The storygeneration module 50 also interacts with the text module 70 to associatethe various story elements 17 with text 72 (see FIG. 43) to compete thestory 19, as further described below.

With respect to building the story 19 to be displayed as a visualrepresentation 18, the process facilitated by the generation module 50can be performed either as a top-down or bottom-up process. The top-downapproach is a user driven methodology in which the story 19 orhypothesis is created by hand in time 402 and space 400, 401. Theanalysts may define the story 19/hypothesis out of thin air with theintent of finding evidence (i.e. provided by the data objects 14) thatsupports or refutes it. The bottom-up approach envisions an analyststarting with raw evidence (data objects 14) and carefully building upthe story 19 that explains a possible scenario. In one example, thescenario may describe a possible threat. This bottom-up process isreferred to as story marshaling—the process by which evidence isassembled into the story 19.

The bottom up approach uses the matching/aggregating of the data intothe data subsets 15. Pattern matching algorithms (e.g. provided by themodule 600, 60) are used to find significant or relevant patterns inlarge, raw data sets (i.e. the data objects 14) and presenting them tothe analyst as story elements 17 within the visual representation 18. Asdiscussed earlier, referring to FIG. 32, the story generation module 50coordinates the performing of the pattern matching using the patterntemplates 59 and/or pattern aggregates 62, as further described below.The pattern assistant module 50 can coordinate the use of algorithmsincluding but not limited to, clustering, pattern recognition, machinelearning or user-drive methods to extract/identify the specific patternsfor assigning to the data subsets 15. For example, the following story19 patterns can be identified and retrieved for specific sequence ofevents 20, such as but not limited to: plot patterns (a sequence ofevents); turning points in plots; plot types; characters and places;force and direction; and warning patterns.

In turn, the module 50 can provide the visualization manager 112 withthe identified story elements 17 (including representations 56 assignedto data subsets 15 extracted from the data objects 14) used to assemblethe story 19 as the visualization representation 18 (see FIG. 33). Inanother embodiment, the module 50 can be used to provide story text 72,generated through interaction with the text module 70 (and userinteractions), to the visualization manager 112, along with the storyfragments associated with the story text 72 as hyperlinked visualizationelements (see FIG. 43), as further described below.

Aggregation Module 600

Referring again to FIG. 32, one step in the process of generating thestory 19 can be through use of the aggregation module 600 for analyzingthe data objects 14 for summarizing and condensing into patternaggregates 62 (see FIGS. 23 and 24). It is recognized that the patternaggregates 62 are a result of identifying possibilities in the raw datafor reducing the data clutter, due to aggregation of similar dataobjects 14 according to such as but not limited to: type; spatialproximity; temporal proximity; association to the same event 20, entity24, location 22; and other predefined filters 602 (see FIG. 22), asdesired. Further, it is recognized that the use of the aggregationmodule 600 is used mainly for data de-cluttering, and as such thepattern aggregates 62 identified are not necessarily for direct use asstory elements 17 until identified as such via the pattern module 60.

In this manner, the amount of data that is represented on the visualinterface 202 can be multiplied. This approach is a way to addressanalysis of massive data. These pattern aggregates 62 can be associatedwith indicators of activity, such as but not limited to: clustering;day/night separation; tracks simplification; combination of similarthings/events; identification of fast movement; and direction ofmovement. For example, a series of email communications over an extendedperiod of time, between two individuals, could be replaced with a singlerepresentative email communication visual connection element 412, thushelping to de-clutter the visualization representation 18 to assist inidentification of the story elements 17.

Referring to FIG. 34, shown is a sketch of raw communication andtracking events (as given by the data objects 14) in time 402 and space400. Referring to FIG. 35, shown is an image of the same data as in FIG.34, but now including pattern aggregates 62 applied using theaggregation module 600 to simplify the diagram and reduce data clutter.In this figure, events have been clustered into days by location andsummary trails, replacing groups of events 20.

It is recognized that the user can alter the degree of aggregation viaaggregation parameters, either automatic (ie. Tool pre-definitions) ormanual (entered via events 109) or a combination thereof. For example,consider the aggregated scenario shown in FIG. 35, having a first degreeof aggregation including pattern aggregates 62 with a ghosted view ofconnections 412 shown in FIG. 34, which is used to denote presence but alesser degree of importance on the individual ghosted connections 412.Therefore, FIG. 35 can represent an entity 24 that may have stopped atseveral different locations before reaching a final destination.

Thus, a group of events 20 may be summarized by the aggregation module600 to show only a representative summarized event 20. Alternatively, auser may wish to aggregate all event 20 objects having a certaincharacteristic or behaviour (as defined by the filters 602—see FIG. 22).

Pattern Module 60

Referring to FIG. 32, the pattern module 60 is used to identify datasubsets 15 that are applicable as story elements 17 for connectingtogether to make the story 19. The pattern module 60 uses predefinedpattern templates 59 to detect these data subsets 15 from the dataobjects 14 and associations 16 making up the domains 400,401,402, eitherfrom scratch or upon review of the de-cluttered data including patternaggregates 62. Accordingly, the pattern module 60 applies the patterntemplates 59 to the data objects 14, associations 16, and/or the patternaggregates 62 to identify the data subsets 15 that are assigned semanticrepresentation 56 to generate the story elements 17.

The pattern module 60 can provide a series of training patterns to theuser that can be used as test patterns to help train the user incustomization of the pattern templates 59 for use in detecting specificpatterns 61 and trends in the data set. The pattern module 60 learnsfrom the training patterns, which can then be used to analyze the dataobjects 14 to provide specific pattern information 61 and trends for thedata objects 14.

For example, referring to FIG. 39, shown is an example pattern template59 for searching the data objects 14, associations 16, and/or thepattern aggregates 62 to identify meeting patterns 61 between two ormore entities 24, further described below. The pattern module 60 appliesthe pattern templates 59 to the data, as well as coordinates the settingof the pattern template 59 parameters, such as type 80 of semanticrepresentation 56, pattern amount, and details 84 of the pattern (e.g.distance and/or time settings). All recognized patterns 61 are thenidentified on the visualization representation 18 in order to contributeto the telling of the story 19.

For example, referring to FIG. 36, the results 61 of pattern template 59matching are shown including aggregated connections 412 and associatedsemantic representations 56. It is also recognized that the thickness ofthe timelines 422 is increased by the template module 60, over thosetimelines 422 of FIGS. 34 and 35, thus denoting evidence ofsummarized/recognized patterns 61. Further, the graph shown in FIG. 36summarizes the events and simply shows the character having traveledfrom a source to a final destination location, with attached semanticrepresentations 56.

Pattern Templates 59

Some examples of pattern templates 59 that could be applied to the dataobjects 14 and associations 16 in order to identify/extract patterns 61are such as but not limited to: activities from data such as phonerecord, credit card transactions, etc used to identify wherehome/work/school is, who are friends/family/new acquaintances, where doentities 24 shop/go on vacation, repeated behaviours/exceptions,increase/decreases in identified activities; and story patterns used toidentify plot patterns (sequence of events 20 such as turning points inplots and plot types, characters 24 and places 22, force and direction,and warning patterns. The pattern templates 59 would be configured usinga predefined set of any of the data objects 14 and/or associations 16 tobe used by the pattern module 60 to be applied against the data underanalysis for constructing the story elements 17.

Pattern Workflow (Detection)

In order to demonstrate integration and workflow of the pattern matchingsystem, two example patterns were developed: a meeting finder patterntemplate 59, and a text search pattern template 59. The meeting finder59 is controlled via a modified layer panel (see FIG. 39), and scans thedata of the memory 102 for conditions where 2 or more entities 24 comewithin a given distance of each other in space and time. The meetingfinder pattern template 59 produces result layers that can be visualizedin numerous ways. The panel allows control of meeting finder algorithmparameters 80,82,84, summary of results, and selection of data paintingtechnique for the results in the scene, further described below. Thetext search pattern template 59 finds results based on string matchescontained in the data, but otherwise works in a similar manner. Itallows a user to search for and identify predetermined patterns withinthe raw data. All identified patterns 61 using the pattern templates 59are then assigned semantic representation(s) 56 via the representationmodule 57, in order to construct the story elements 17 further describedbelow.

Referring to FIG. 40, application of the meeting finder pattern template59 applied to vehicle tracking data shows an identified pattern 88outlined in order to annotate the results of the pattern matching.Accordingly, a potential meeting between two or more entities wasdetected when the parameters 80,82,84 of the pattern template 59 wasapplied against the data of the domains 400,401,402.

Ultimately, the output of the pattern matching is a summarization ofevidence into data subsets 15 with semantic relevance to the story 19.In the visualization of FIG. 40, the identified pattern 88 is an exampleof a data subset 15 suitable for association with a semanticrepresentation (e.g. meeting between John and Frank) to incorporate theidentified pattern 88 as one of the story elements 17 of the resultantstory 19 shown on the visual interface 202. Examples of otheridentifiable patterns are; phone call sequences, acceleration anddeceleration, pauses, clusters etc. Advanced pattern recognitiontemplates 59 may be able to discover other relevant or specializedbehaviors in data, such as “going shopping” or “picking up the kids atschool”, or even plots and deception. It will be understood by thoseskilled in the art that other pattern detection and identificationmethods known in the art such as event sequence and semantic patterndetection may be used either as a standalone or in combination withabove mentioned pattern templates 59, as desired.

Semantic Representation Module 57

The semantic representation module 57 facilitates the assigning ofpredefined semantic representations 56 (manually and/or automatically)to summarized behaviours/patterns 61 in time and space identified in theraw data, through operation of the pattern module 60 and/or theaggregation module 600. The patterns 61 are comprised of data subsets 15identified from the larger data set (e.g. objects 14 and associations16) of the domains 400,401,402). Assigning of predefined semanticrepresentations 56 to the identified data subsets 15 results ingeneration of the story elements 17 that are part of the overall story19 (e.g. a series of connectable story elements 17). The identifiedpatterns 61 can then be visually represented by descriptive graphics ofthe semantic representation 56, as further described below.

For example, if a person is shown traveling a certain route every singleday to work, this repetitive behaviour can be summarized using theassigned semantic representation 56 “daily workplace route” asdescriptive text and/or suitable image positioned adjacent theidentified pattern 61 on the visualization representation. The semanticrepresentation module 57 can be configured to appropriatelyselect/assign and/or position the semantic representation 56 adjacent tothe data subset 15, thus creating the respective story element 17.

Referring now to FIGS. 37 a and 37 b, shown is an exemplary operation ofthe semantics representations 56 applied to the data objects 14. Aperson 24 has traveled from a first location A to a destination locationD, identified as matching a travel pattern template 59 (e.g. sequentialstops from starting point to end destination), and thus assigned as datasubset 15. The person 24 may have stopped at several different locations22 (locations B, C) on route to the destination. Depending upon thesettings within the pattern module 60 (i.e. the amount of detail thatthe user may request to view on the visual representation 18), thepattern module 60 can filter the sequence of events 20 relating tostopping at location B and location C. Thus, as shown in FIG. 37 b, thesemantic representations 56 include a reduction in the amount of datashown, thus portraying a summary of the stream of events (i.e. travelfrom location A to D) without including each event 20 in between, toprovide the story element 17. Further, the semantics representation 56could be used to indicate the specific pattern 60 defining that theperson 24 went from home to church (when traveling from location A toD). Thus, based on the specific pattern information 61, the data subset15 is assigned by the module 57 the semantic representations 56 showinga home marker and a church marker at locations A and D respectively.

It is recognized that the pattern module 60, the semantic representationmodule 57 can operate with the help of the aggregation module 600 inhelping to de-clutter identified patterns 61 for representation as partof the story 19 as the story elements 17, as desired.

Semantics Representation 56

The first step of working at the story level is to represent basicelements such as threads and behaviors with semantic representations 56in time 402 and space 400. For example, suppose one has evidence (ie.raw data objects 14) that a person 24 spends every night at a particularlocation 22, which is recognized as a specific pattern 61. The visualrepresentation 18 of this pattern 61 might include a marker (ie.semantic representation 56) at that location 22 and a hypothesis aboutthe meaning of that evidence that says “this person lives at thislocation” such that the story 19 is associated with the semanticrepresentation 56. An image of a house or a visual element 410 couldalso be displayed in the visual representation 18 to supportunderstanding. The visual element 410 of the home, in this case, istherefore may be an aggregation in space and time of some amount ofevidence as represented in the visual representation 18 as the semanticrepresentation 56 (ie. home marker).

Further, it is recognized that threads in the story 19 can be explicitlyidentified through operation of the story generation module 50.Respective threads can be defined (by the user and/or by configurationof the tool 12 using data object 14 and association 16 attributes) as agrouping of selected story elements 17 that have one or more commonproperties/features of the information that they relate to, with respectto the overall story 19. Accordingly, the story fragments/elements 17 ofthe story 19 can be assigned (e.g. automatically and/or manually) to oneor more thread categories 910 (see FIG. 45) with an associatedrespective color (or transparency setting, label, or other visuallydistinguishing feature) for visual identification in the story 19, asdisplayed in the visualization representation 18. The visibility ofthese thread categories 910 can be toggled, e.g. as a parameter 911(e.g. filter) for configuring the display of the story 19 on the visualinterface 202, to allow the user to focus on a subset of the story 19,as desired. The associated visual distinguishing parameter 911 for thethread categories 910 can facilitate at-a-glance identification by theuser of the thread categories 910 and the story elements 17 theycontain. It is also recognized that use of the thread categories 910facilitates the user to select specific data subsets (from the overalldata set of the story 19) to concentrate on during data analysis.

Thus, in operation, the semantic representations 56 can be used toreduce the complexity of the visual representation 18 and/or tootherwise attach semantic meaning to the identified patterns 61 toconstruct the story 19 as the series of connected story elements 17. Inone aspect, the semantic representations 56 are user defined for aspecific pattern 61 or behaviour, and replace the data objects 14 withan equivalent visual element that depicts meaning to the entity 24 andevents 20.

As mentioned earlier, in one aspect, the semantics representation 56 canbe user entered such that a user may recognize a specific pattern 61 orbehaviour and replace that pattern with a specific statement orgraphical icon to simplify the notation used by the pattern module 60.Alternatively, the semantics representation 56 can be stored within apattern templates 59 that is in communication with the pattern module60, such that all occurrences of the desired pattern 61 are found andreplaced by the semantic representation 56 in the spatial-temporaldomains 400,401,402.

Referring to FIG. 41, shown are four example visualization paints (e.g.semantic representations 56) applied to the same identified datapatterns 61. Rubber-band 90, Bezier 92, Arrows 94, and Coloured 96 Notethat these qualities can be combined, as desired. Other qualities suchas text, size, and translucency can also be altered, as desired. Thetechnique for visualizing of the identified/detected results of thepattern matching (e.g. patterns 61) can be referred to as a datapainting system. It enables visualization rendering techniques to beattached to pattern 61 results dynamically. By decoupling thevisualization technique (e.g. semantic representations 56) from thepatterns 61 in this way, the pattern recognition stage only needs tofocus on the design of pattern matching templates 59 for the specificattributes of the data objects 14 to match, rather than bothvisualization of the identified patterns 61 and the pattern matchingitself. Further, the pattern 61 detection may be either completely orpartially user-aided. It will be understood by a person skilled in theart that these visuals (e.g. visualization parameters assigned toaspects of the detected pattern) can be easily extended and married toexisting and future patterns or templates.

Referring to FIG. 42, shown are example of numerous semanticrepresentations 56 applied to pattern 61 results that are used toidentify story elements 17 of the story 19. The story shown representsthe passing of information in a planned assassination by two parties.

Text Module 70

Referring again to FIGS. 32 and 43, developing a system for presentingthe results of pattern analysis in the form of a story that can be“told” in the context of time and space is a key research objective. Ifthe entities 24 and events 20 of the data objects 14 representcharacters and events in the story 19, and the space-time view is like asetting, then a method by which an author orders and narrates a sequenceof views to present to others can be done. View capturing is a basiccapability of the story generation module 50 for saving perspectives intime and space, and can be used to recall key events or aspects of thedata. This system has been extended to allow the analyst to author asequence of saved views 95 linked to a text explanation 72 via links 96.

This FIG. 43 shows the story 19 narration concept. The captured views 95appear along the bottom of the visualization representation 18 asthumbnails, for example. These thumbnails can be dragged into thetextual elements 72 and can be automatically linked, for example.Subsequently, upon review of the story text 72, the analyst can click onthe link 96 to have the selected scene/view 95 recreated on the visualinterface 202 (e.g. using the saved parameters of the included data—suchas filter settings, selected groupings 27 of objects 14, navigationsettings, thread categories 910, and other visualization representation18 and story 19 view setting parameters as describe above). It isrecognised that for the recreated scene/view 95 embodiment, furthernavigation and/or modification of the recreated view would be availableto the user via user events 109 (e.g. dynamic interaction capabilities).It is also recognised that the captured views 95 could be saved as astatic image/picture, which therefore may not be suitable for furthernavigation of the image/picture contents, as desired.

The text navigator, or power text, module 70 allows the analyst to writethe story 19 as story text 72 and embed captured views 95 directly intothe text 72 via links 96. The views 95 capture maintains all of theinformation needed to recall a particular view in time and space, aswell as the data that was visible in the view (including patternvisualizations where appropriate). This allows for an authoredexploration of the information with bookmarks to the settings.Additionally, this allows for a chronotopic arrangement to the elements17 of the story 19. The reader can recall regions of time that arerelevant to the narrative instead of the order that things actuallyhappened.

In one embodiment, the user first navigates the visualizationrepresentation 18 to a selected scene. To link a new view into to thestory text 72, the analyst clicks a capture view button of the userinterface 202. A thumbnail view 95 of the scene can be dragged into thestory text 72, automatically linking it into the power text narrative.The linkage 96 can include storage of the navigation parameters so thatthe scene can be reproduced as a subset of the complete visualizationrepresentation 18. When the analyst clicks on the view hyperlink 96, thetool 12 redisplays the entire scene that was captured. The analyst atthis point is free to interact with the displayed scene or continuereading the narrative of the story text 72, as desired. This storytelling framework (combination of story text 72 and captured views 95)could even be automated by using voice synthesizers to read the storytext 72 and recall the setting sequence.

The power text system also supports a concept of story templates 71 (seeFIG. 32) that include predefined segments of the story text 72, whichcan be further modified by the user. These story templates 71 can bepredetermined sections or chapters in the story 19, which can serve toguide generation of the story 19 content. For example, an incidentreport template 71 might contain headings for “Incident Description”,“Prior History of Perpetrator” and “Incident Response”. Another optionis for the predefined segments of the story text 72 to be part of thestory 19 content, and to provide the user the option to link a selectedview 95 thereto. For example, one of the predefined segments in a battlestory template 71 could be “Location of battle A included armed forcesresources B with casualty results C, [link]”. The user would replace thegeneric markers A,B,C with the battle specific details (e.g. furtherstory text 72) as well as attach a representative view 95 to replace thelink marker [link]. Accordingly, the story templates 71 could be used toguide the user in providing the desire content for the story 19,including specific story text 72 and/or captured views 95.

The power text module 70 focuses on interactive media linking. The views95 that are captured can allow for manipulation and exploration oncerecalled. It will be understood that although a picture of the capturedview 95 has been shown as a method of indexing the desired scene andcreating a hyperlink 96, other measures such as descriptive text orother simplified graphical representations (e.g. labeled icon) may beused. This is analogous to a pop-up book in which a story 19 may beexplored linearly but at any time the reader may participate with thecontent by “pulling the tabs” if further clarity and detail is needed.The story text 72 is illuminated by the visuals and the content furtherunderstood through on-demand interaction.

Referring to FIG. 44, shown is a further embodiment of stories workflowprocess 900. The workflow process comprises story building 901 and storytelling 903.

At step 902, raw data for visualization representation 18 is received.At step 904, the raw data objects 14, comprising a collection of events(event objects 20), locations (location objects 26) and entities (entityobjects 24) is applied to a pattern module 60. For example, as shown inFIG. 39, the meeting finder pattern template 59 can be used to searchfor and display patterns 61 in raw data (i.e. by finding events thatoccur in close proximity in time and space). Alternatively, othertechniques mentioned earlier such as text searching, residence finder,velocity finder and frequency analysis might be used to identify certainpatterns or trends 61 in the data objects 14. It will be understood thatthe above-mentioned pattern detection techniques may be used as astand-alone or in combination with known pattern identification methods.

The visualization tool 12 has a data painting system (or othervisualization generation system) described earlier then uses the patternresults 61 provided by the pattern identification at step 904 to applynumerous graphical visualizations (e.g. representation 56) to selectedfeatures of the pattern results 61. Various visualization parameters forthe pattern 61 can be altered such as its text, size, connectivity type,and other annotations. The system for visualizing the identified patternas defined by step 906 can be partially or completely user aided.

At step 908, a user can create a story 19 made up of text 72 andbookmarked views of a scene. The bookmarked views are created at step910 and may be shown as thumbnails 95 depicting a static picture of acaptured view. The hyperlinks 96, when selected, allow a user todynamically navigate the captured view or scene (as a subset of thevisualization representation 18). For example, they may provide theability to edit the scene or create further scenes (e.g. changeconfiguration of included data objects 14, add/remove data objects 14,add annotations, etc.). Each captured view at step 910 would comprise ofa scene depicting the entities, locations and corresponding events in aspace-time view as well as applied graphical visualizations. Further,templates 71 can be created/modified using certain portions of the story19, which includes previously captured hyperlinks 96. These templates 71can be stored to the storage 102 and can then be used to apply to othersets of data objects 14 to write other stories 19 as part of the storytelling process 903.

Other Components

Referring again to FIG. 32, the visualization tool 12 has avisualization manager 112 for interacting with the data objects 14 forpresentation to the visual interface 202 via the visualization renderer112. The data module 114 comprises data objects 14, associations data 16defining the association between the data objects 14 and pattern data 58defining the pattern between data objects 14. The data objects 14further comprise events objects 20, entity objects 24, location objects22. The data objects 14 can then be formed into groups 27 throughpredefined or user-entered association information 16. The user enteredassociation information 16 can be obtained through interaction of theuser directly with selected data objects 14 and association sets 16 viathe time slider and other controls shown in FIG. 3. Further, thepredefined groups 27 could also be loaded into memory 102 via thecomputer readable medium 46 shown in FIG. 2. Use of the groups 27 issuch that subsets of the objects 14 can be selected and grouped throughthe associations data 16.

The data manager 114 can receive requests for storing, retrieving,amending or creating the data objects 14, the associations data 16, orthe data 58 via the visualization tool 12 or directly via from thevisualization renderer 112. Accordingly, the visualization tool 12 andmanagers 112, 114 coordinate the processing of data objects 14,association set 16, user events 109, and the module 50 with respect tothe content of the visual representation 18 displayed in the visualinterface 202. The visualization renderer 112 processes the translationfrom raw data objects 14 and provides the visual representation 18according to the pattern information 61 provided by the pattern module60.

Note that the operation of the visualization tool 12 and the storygeneration module 50 could also be applied to diagram-based contextshaving a diagrammatic context space 401. Such diagram-based contextscould include for example, process views, organization charts,infrastructure diagrams, social network diagrams, etc. In this way, thevisualization tool 12 can display diagrams in the x-y plane and showevents, communications, tracks and other evidence in the temporal axis.For example, in a similar operation as described above, story generationmodule 50 could be used to determine patterns 61 within the data objects14 of a process diagram and the visual connection elements 412 withinthe process diagram could be aggregated and summarized using theaggregation module 600 and the pattern module 60 respectively. Thesemantics representation 56 could also be used to replace specificpatterns 61 within the process flow diagram.

The visualization tool 12, as described can then use simple queries orclustering algorithms to find patterns 61 within a set of data objects14. Ultimately the output of the story generation module 50 or auser-driven story marshaling is an aggregation of evidence into a groupwith semantic relevance to the story 19.

Generation of the Story 19

Thus, the representation of the story 19 begins with the representationof the elements from which is it composed. As discussed earlier, thereare 3 visual elements that are designed to support the display ofstories 19 in the visualization tool 12:

1. Story Fragments 17: Aggregate Event Representation 62

-   -   Summarize a group of events 20 with an expression in time 402        and space 400. Allow aggregates 62 to be aggregated further;

2. Visual association of identified data subsets 15 as story elements 17to the Story 19

-   -   Express where and how elements 17 and thread categories 910        (e.g. groupings of selected threads) connect and interact        (discussed relating to FIG. 38); and

3. Annotation of Semantic Meaning 56

-   -   Iconic, textual, or other visual means to convey importance or        relevance to the story. This can involve user participation        and/or some automated means (through the use of pattern        templates 59 detecting specific patterns 60 and replacing the        patterns 60 with predefined semantic representations 56).

Referring now to FIG. 38, shown is an exemplary process 380 of thevisualization tool 12 when processing new story elements 17 of evidence(as identified from the data objects 14 of the domains 400,401,402). Atstep 382, the new story elements 17 of evidence are selected forcorrelation with the existing story 19 using the story generation module50. If specific patterns 61 are found within the evidence at step 384,the patterns 61 can then be assigned the semantic representation 56using the module 57 at step 386, in order to create the story element17. Optionally, at step 30 the text module 70 can be used to insert/linkthe story element 17 into story text 72.

Further, it is recognized that output of the story 19 could be saved asa story document (e.g. as a multimedia file) in the storage 102 and/orexported from the tool 12 to a third party system (not shown) over thenetwork, for example, for subsequent viewing by other parties. It isrecognized that viewing of the story 19, once composed and/or duringcreation, can be viewed as an interactive movie or slideshow on thedisplay. It is also recognized that the story document could also beconfigured for viewing as an interactive movie or slideshow, forexample. It is recognized that the format of the story document can bedone either natively in the tool 12 format, or it can be exported tovarious formats (mpg, avi, powerpoint, etc).

It is understood that the operation of the visualization tool 12 asdescribed above with respect to the stories 19 can be implemented by oneor more cooperating modules/managers of the visualization tool 12, asshown by example in FIG. 32.

Timeline Bar and Focus Bar

The visualization interface 202 may also include a timeline bar 840 andan adjacent/coupled focus bar 850 as shown in FIGS. 1, 46 a, 46 b, 46 c,46 d, 46 e, 46 f. The bars 840, 850 allow the user to navigate andscroll through data objects 14 and/or associations 16 displayed in thevisualization representation 18 (see FIG. 1). Use of the bars 840, 850and their various controls, as further described below, provide foradjustments in the visible objects 14, associations 16, and thecorresponding time range (e.g. focus range 844) of the temporal domainshown in the visualization representation 18. The use of the bars840,850 provides for the user to focus on specific periods of time, zoomin on particular sequences of objects 14—associations 16, or to watchevents unfold in an animation.

Referring to FIG. 46 g, the time bar 840 properties (e.g. scale,selected limits 837,838,841,843, focus slider 842 operation, etc.) iscoordinated/defined/implements by a time module 112 a and the focus bar850 properties (scale, associated chart 890, extents—e.g. 841,843—,etc.) is coordinated/defined/implemented by a focus module 112 b of theVI module 112, in association with any other herein described modules ofthe tool 12 and/or the user events 109.

The timeline bar 840 and focus bar 850 are time scales (e.g. linear,logarithmic) that are visible on the side of the visual representation18. It will be appreciated, however, that these bars 840, 850 could alsobe located above, below or on either side of the visual representation18, or on any other portion of the user interface 202. A user caninteract with the timeline bar 840 to set a focus range 844 of a totalavailable time period 839 (e.g. having defined global time/temporallimits 837,838 displayed—or otherwise—of the time bar 840), whichdefines the temporal range of the data objects 14 and/or associations 16that may be displayed (e.g. selected from the dataset of available dataobjects 14 and associations 16 in the specified temporal domain) in thevisualization representation 18. The focus bar 844 includes an assignedlocal future/start data limit 841 (e.g. selectable by the user), anassigned local past/end data limit 843 (e.g. selectable by the user).The time bar 840 includes a focus slider 842 having the same locallimits 841,843 (e.g. having similar time extents as the focus range 844)that represents a defined/selected window/block of time (e.g. a timerange) that can be moved/manipulated along the timeline bar 840 havingthe defined total temporal global extents of times 837,838, a startfocus time control 848 that is used to set one side of the temporallocal limit of the focus slider 842, a end focus time control 846 thatis used to set the other side of the temporal local limit of the focusslider 842, and time indicators 847 that represent units of measure fortime along the timeline bar 840 (e.g. having an axis 845). Together, thestart local limit 841 and the end local limit 843 are used by thevisualization tool 12 to define the entire set of data objects 14 and/orassociations 16 that are potentially viewable (with manipulability) inthe visualization representation 18 by the user via interaction with theuser interface 202, and as such the local limits 841,843 define thetemporal boundaries of the displayed data objects 14 and/or associations16 (i.e. those objects 14 and/or associations 16 having temporalattributes that are outside of the local limits 841,843 would not beavailable for viewing in the visualization representation 18).

Accordingly, it is recognized that the focus bar 850, as defined havingthe same time local limits 841,843 as the focus range 844, is used bythe visualization tool 12 to select a subset of data objects and/orassociations 16 from the total (those data objects and/or associations16 associated with temporal attribute(s) included in the temporal domainbetween the defined global limits 837,838) for display as a subset inthe visualization representation 18.

The bars 840,850 together are used in the visualization representation18 to represent a temporal coordinate system defined by the axis 845,providing a reference dimension of temporal measurement through unitlengths (i.e. time indicators 856 of the focus bar 850), which aremarked off along the time axis 845 (e.g. using equidistance intervals torepresent a linear time scale of the temporal domain). The timeline bar840 is in a dynamic coupled relationship with the focus bar 850 and thevisualization representation 18, such that the focus bar 850 contains anexpanded time range defined by the focus range 844 (i.e. the start andend times—extents—of the focus bar 850 can be the same as the start andend times—extents—of the focus range 844) of the timeline bar 840.

For example, the unit distance (e.g. scale) between the time indicators856 is larger than the unit distance between the time indicators 847and/or there are a fewer number of time indicators 856 as compared tothe time indicators 847, i.e. the temporal scale provides a “shorthand”form for discussing relative temporal lengths between adjacentindicators 847,856.

In interaction with the bar(s) 840, 850 by the user (e.g. mouse clicks,keyboard entries, etc. through user events 109) directs a module toinstruct, for example, the Visualization manager 300 to communicate withthe data manager 114 for generating a subset of data objects 14 and/orassociations 16 that have temporal attributes that are within the focusrange 844 (i.e. defined range of time between local limits 841,843).Alternatively, the manager 300 may instruct another module (such as theAggregation Module) to generate a subset of the data objects 14 and/orassociations 16. The Visualization manager 300 receives the subset ofdata objects 14 and/or associations 16 from the Data manager 114 (or theAggregation Manger of the Aggregation Module 600) and uses, generates orupdates the visualization components 308 (e.g. sprites). The manager 300communicates the sprites 308 to the VI Manager 112. The VI Manager 112renders the sprites 308 to create the final image including visualelements representing the data objects 14 and/or associations 16 fordisplay in the visual representation 18 on the interface 202.

Referring again to FIG. 46 b, the expanded time scale of the focus bar850 (having the local limits 841,843 for defining the temporal range ofthe focus bar 850), as compared to the time scale of the timeline bar840 (having the global limits 838,838 for defining the total temporalrange 839 of the time bar 840, such that the defined total temporalrange 839 is greater than the defined focus range 844 and the definedfocus range 844 is positioned within the defined total temporal range839), is presented as an increase in the distance between similarmeasurement unit markings displayed on the user interface 202. Forexample, the distance on the user interface 202 between the measurementunits of 10 AM and 11 AM in focus range 844 of the timeline bar 840 isless that the distance between corresponding measurement units of 10 AMand 11 AM in the focus bar 850. The measurement units can be referred toas time indicators 847,856, which represent a point in the temporaldomain and the space (i.e. distance along the time axis) between timeindicators 847,856 represents the period of time in the temporal domainbetween time indicators 847,856.

Accordingly, a user can focus on a narrower set of data objects 14and/or associations 16 (i.e. potentially available having a temporalattribute between the local limits 841,843) by interacting with thefocus slider 842 to set the focus range 844. A user may, for example,only wish to view data objects 14 and/or associations 16 from a startperiod t1 to an end period t2 (i.e. the focus range 844). A user can,for example, select the past/start focus time control 848 with the mouseand drag it to a selected time indicator 847 which corresponds to t1.Likewise, the user can select the future/end focus time control 846 anddrag it to a selected time indicator 847 which corresponds to t2,thereby defining the extent (e.g. time range/window) of the focus slider842.

Focus Bar 850

As described above, the focus bar 850 is an expanded view of the focusrange 844 that is selected/defined on the timeline bar 840. The focusbar 850 includes a past focus time 852 and a future focus time 854 thatare defined as the time range (i.e. limits 841,843), set by the focusrange 844, focus time indicators 856 that define the time units ofmeasure (e.g. equidistant) and a moment of interest control 858 forrepresenting the location of the reference surface 404. TheVisualization manager 300 can receive user events 109 as the userinteracts with the past focus time control 848 and the future focus timecontrol 846. The Visualization manager instructs the VI manager 112 toredraw the focus bar 850 and to set the past focus time 852 to the timeindicator 847 that corresponds to the past focus time control 848 and toset the future focus time 852 to the time indicator 847 that correspondsto the future focus time control 846. The VI manager renders a new focusbar 850 to the user interface 202 each time the user interacts with thecontrols 846,848 of the timeline bar 840, such that the time indicators856 are distributed on the timescale 845 of the focus bar 850 betweenthe limits 841,843.

The moment of interest control 858 is a marking used to represent thepoint in the temporal domain at which the reference surface 404 of thevisualization representation 18 is located. The moment of interestcontrol 858 can be located anywhere along the focus bar 850, for exampleat one of the extents of the focus bar 850 such as the focus time 852.The moment of interest control 858 represents the temporal state of theobjects 14 and/or associations 16 present with respect to the referencesurface 404. The instant of focus control 858 can be a primary temporalcontrol of the visualization representation 18. It can be adjusted bydragging it up or down with the mouse pointer across the focus bar 850to the desired position. The instant of focus (also known as the browsetime) is the moment in time represented at the reference surface 404 inthe spatial-temporal visualization representation 18. As the instant offocus control 858 is moved by the user forward or back in time along thefocus bar 850 the display of the visualization representation 18 isupdated. For example, the placement of event visual elements 14 animatealong the timelines and entity visual elements 14 move along thereference surface 404 interpolating between known location visualelements (see FIGS. 6 and 7). Examples of movement of the instant offocus 858 are given with reference to FIGS. 14, 15 and 16 above. It isrecognized, as discussed above, the that focus control 858 can bepositioned at one of the extents (e.g. focus times 852,854) andtherefore the position of the focus control 858 in the temporal domainis adjusted as the focus slider 842 is manipulated/moved along thetimeline bar 840.

Operation of the Bars 840, 850

As mentioned above, interaction with the timeline bar 840 through userevents 109 provides for dynamic updates to the visualizationrepresentation 18 via the VI manager 112. For example, a user may dragthe past focus time control 848 back (e.g. to a different location alongthe time bar 840) in time (i.e. towards the past data limit 843) therebyincreasing the focus range 844. This causes the past focus time 841 ofthe focus bar 850 to be adjusted to the same time as the adjusted timesetting of the past focus time control 848 (e.g. setting the control 848to 12:30 pm would cause the time module to set the focus time 841 alsoto 12:30 pm). In turn, the display of the objects 14 and/or associations16 in the visualization representation 18 will be updated to reflect thechange in the discussed temporal boundary, as commensurate with thenewly defined limits 841,843 also populated/assigned to the focus bar850. It will be appreciated that the visualization representation 18could display additional visual elements representing additional dataobjects 14 and/or associations 16 because the temporal range representedin the visualization representation 18 (i.e. the focus range 844) is nowlarger. Alternatively, the user may decrease the focus range 844 bymoving the past focus time control 848 forward in time (i.e. towards thefuture data limit 841) and/or the future focus time control 846 backwardin time (i.e. towards the past data limit 843). Each time the user moveseither the past focus time control 848 or the future focus time control846, the corresponding limits of the focus bar 850, namely the focustimes 854, 852 are also adjusted to match, and as well the correspondingnew visualization representation 18 is generated. An example of changingthe controls 846,848 is shown in FIG. 46 c.

Also in FIG. 46 c, it should be noted that the distance between themeasurement units (e.g. time indicators 856) adjusts accordingly toreflect the changes in the focus times 841,843. It is recognized thatthe length of the axis 845 of the focus bar 850 can remain constantwhile the changes in the focus times 841,843 and associated distancebetween the measurement units (e.g. time indicators 856) is coordinatedby the time module (of the visualization tool 12—see FIG. 1). It is alsorecognized that the length of the axis 845 of the timeline bar 840remains constant during adjustment of the controls 846,848, as does theassociated distance (i.e. remain constant) between the measurement units(e.g. time indicators 847) of the timeline bar 840, as the extents ofthe focus slider 842 are widened/narrowed.

Also, referring to FIG. 46 d, the user has an option of dynamicallyscrolling through the entire data set (i.e. the data objects 14 acrossthe entire temporal range available defined by the past data limit 837and the future data limit 837) using a fixed focus range 844. The fixedfocus range 844 acts a window/defined temporal block in the temporaldomain in which to view the movement, activity and other characteristicsassociated with data objects 14 and/or associations 16. The user canselect the focus slider 844 with a mouse cursor 847 and move it up anddown along the timeline bar 840. Updates to the contents ofvisualization representation 18 will be generated on the user interface202 as the focus slider 844 is moved. It is also noted in FIG. 46 d thatas the focus slider 842 is translated along the timeline bar 840, theeffect of scrolling though time is presented in the focus bar 850 as thetime indicator 856 contents of the focus bar 850 are adjusted/updated tomatch the changes in the focus times 841,843 driven through movement ofthe focus slider 842. In this example, the extent of the focus slider842 remains constant as the focus slider 842 is translated/displacedalong the timeline bar 840. Continuous animation of data objects 14and/or associations 16 over time and geography is therefore provided inthe representation 18 as the focus slider 842 is moved forward andbackward in time on the timeline bar 840.

Referring to FIG. 46 e, the displayed temporal scale of thevisualization representation 18 can be adjusted independently from thebars 840,850 through manipulation of a scale control 859, for examplelocated on the axis 845.

It will be appreciated that the instant of focus 858 may represent anyinstant of time that is of interest to the user in the temporal domain.For example, the instant of focus may represent a central period wherebydata elements 20, 21, 22, 23, 24, 26 are shown above or below thereference surface 404 depending on their corresponding temporalproperty. A user, however, may choose the instant of focus to be astarting time (i.e. the past focus time 841), so that data elements 20,21, 22, 23, 24, 26 are shown on or above the reference surface 404.

Analysis Module 1000

Referring to FIG. 48, the Analysis Module 1000 is for, such as but notlimited to, summarizing a subset of data objects 14 based onuser-defined criteria, and providing the subset of data objects 14 as aninteractive visual result list 1028, 1034, 1046, 1054, 1068, 1074, 1086(see FIG. 49 a) to the VI Manager 112 for rendering to the visualinterface 202. It is also recognized that the data in the result lists1028, 1034, 1046, 1054, 1068, 1074, 1086 can be displayed in thevisualization representation 18 in the form of a selected chart type(e.g. summary chart 890—including for example table output 876 and/orchart output 886—and/or count charts 1208).

User events 109 that are input via interactions with the analysiscontrol functions/operators 1020, 1030, 1040, 1050, 1060, 1070 and 1080(see FIG. 49 a), as part of the interface controls 306, are processed bythe Visualization manager 300. Each analysis control function 1020,1030, 1040, 1050, 1060, 1070 and 1080 can be associated with individualrespective managers/modules 1004, 1006, 1008, 1010, 1012, 1014 and 1016.As described below by example, the Analysis Module 1000 is extendibleand may have any number of analysis control functions each of which canbe associated with a respective manager/module. The analysis controlfunctions may be pre-defined in the tool 12 and/or added dynamically bythe user via interaction with the user interface 202. As the userinteracts with an analysis control module 1000, the Visualizationmanager 300 communicates with the Analysis Manager 1002 which in turnissues instructions to the manager that is associated with the controlthe user is interacting with, for example. For example, when the userinteracts with the Gap Finder control 1080 the Visualization manager 300communicates with the Analysis manager 1002, which instructs the GapFinder Manager 1014 to create a visual result list 1086.

It is also recognized that the user can interact directly via theinterface 202 with the desired control module 1004, 1006, 1008, 1010,1012, 1014 and 1016 of the analysis module 1000, as desired. Anadvantage of the using any of the analysis control functions 1020, 1030,1040, 1050, 1060, 1070 and 1080 is that the respective results list(s)1028, 1034, 1046, 1054, 1068, 1074, 1086 can be output in chart form(e.g. summary chart 890, count chart 1208, etc.) and displayed inassociation with the timeline and/or focus bar(s) 840,850 (see FIGS. 46a and 46 f. For example, each of the entries 891 in the summary chart890 of FIG. 46 f are assigned to a respective (one or more) timeindicators 856 of the focus bar 850. It is also recognized that each ofthe entries in the summary chart 890 of FIG. 46 f can be assigned to arespective (one or more) time indicators 847 of the time bar 840, as analternative or in addition to the indicators 856 as shown by example.

Meeting Finder Control 1020

Referring to FIG. 49 a, the meeting finder control 1020 enables a userto find meetings (i.e. a type of event 20) between entities 24 that arewithin a selected distance in the spatial domain and a selected range oftime in the temporal domain. The meeting finder control 1020 includes adistance setting 1022, a range setting 1024, execute data object 14selections 1026 a and 1026 b, and result list 1028. The result list 1028includes identified meetings 1029 that are within the event/meetingsearch parameters defined by the distance setting 1022 and the rangesetting 1024. In operation, a user sets the search parameters 1022,1024by interacting with the distance setting selector 1022 and the rangesetting selector 1024. The user can choose to execute the search againstall of the entities 24 in the visual representation 18 or a subset (e.g.group) of entities 24 that have been selected by the user from thevisual representation 18 of the user interface 202. It is alsorecognized that the meetings 1029 can be included in chart information(e.g. as meeting counts for each entity 24).

Interactions with the execute functions 1026 a,1026 b are handled by theVisualization manager 300. The Visualization manager 300 instructs theAnalysis Tool Manager 1002 to retrieve the subset of data objects 14that is within the search parameters 1026 a,1026 b. The Analysis ToolManager 1002 communicates the request to the Meeting Finder Manager 1004which queries the Data Manager 114. The Data Manager 114 provides a datasubset to the Meeting Finder Manager 1004 which formulates the data asan output 1005. Finally, the Analysis Tool Manager 1002 instructs the VIManager 112 to render the output 1005 as an interactive visual resultlist 1028 on the user interface 202.

Referring to FIG. 49 b, the Meeting Finder control 1020 findsoccurrences where selected 1026 a,b Entities 24 are co-located within auser-specified space and time ranges 1022,1024. For example, if twoEntities 24 have any events 20 that are less than the set time spanapart, AND those same events 20 are less than the set distance apart,those two events 20 will be considered a meeting 20 and placed in theresult list 1028. If multiple meetings 20 share the same events 20, theywill be merged into one large meeting 20 (e.g. a result of eventaggregation as further described herein).

Gap Finder

Referring to FIG. 49 a, the Gap Finder Control 1080 enables a user tofind reporting gaps 1088 of entities 24 which are greater than aspecified amount of time 1082 in the temporal domain. Reporting gaps1088 are temporal periods in which there is no occurrence of an entity24 in the visual representation 18. The Gap Finder Control 1080 includesa range selector 1082, selection functions for data object 14/entity 24selection 1084 a, 1084 b and result list 1086. Result list 1086 ispopulated with the gaps 1088 that are within the search parameter 1082when the user selects one of the selection functions 1084 a, 1084 b.

Interactions with the functions 1084 a,1084 b are handled by theVisualization manager 300. In an example operation of the control 1080,the Visualization manager 300 instructs the Analysis Manager 1002 toretrieve the subset of data objects 14 that is within the searchparameters 1082 1084 a,b. The Analysis Manager 1002 communicates therequest to the Gap Finder Manager 1014 which queries the Data Manager114. The Data Manager 114 provides the data subset to the Gap FinderManager 1002 which formulates the data as an output 1015. The AnalysisManager 1002 then instructs the VI Manager 112 to render the output 1015as an interactive visual result list 1086 on the user interface 202(e.g. in chart form 890, 1208 in the visual representation 18).

Accordingly, in view of the above, and in reference to FIG. 49 c, theGap Finder 1080 searches and reports on time gaps 1088 in an entities 24event 20 history of the visual representation 18. The minimum time span1082 of the potential gap(s) can be set by the analyst. The gap finder1080 will find gaps 1088 in reporting (e.g. in the information of thedata objects 14) greater than the set time value, for example.

Speed Finder

Referring to FIG. 49 a, the Speed Finder Control 1060 enables a user tofind and highlight entity 24 movement intervals in a results list 1068where an interpolated velocity of the interval is between a lower speed1062 and an upper speed 1064. Speed is the rate of change of position inthe spatial domain, and is expressed in units of distance divided bytime. The interpolated speed can be the average speed of an entity 24when travelling between two positions in the spatial domain. It isrecognized that the results list 1068 can also indicate where theidentified entity 24 has exceeded the maximum/upper speed 1064, forexample. The Speed Finder Control 1060 includes a lower speed parameter1062 selection, an upper speed parameter 1064 selection, entity 24selection functions 1066 a, 1066 b, and result list 1068. Together, theupper speed 1064 and the lower speed 1062 define some of the searchparameters. The results list 1068 can include entities 24 that arewithin the search parameters 1062, 1064 (e.g. entities 24 that aretravelling at an interpolated speed between the lower speed 1062 and theupper speed 1064).

In an example operation of the module 1080, interactions with selectionfunctions 1066 a, 1066 b are handled by the Visualization Manager 300.The Visualization Manager 300 instructs the Analysis Manager 1002 toretrieve the subset of data objects 14 that corresponds to the searchparameters 1062, 1064. The Analysis Manager 1002 communicates therequest to the Speed Finder Manager 1010 which queries the Data Manager114 with the search parameters 1062, 1064. The Data Manager 114 providesthe data subset to the Speed Finder Manager 1010 which formulates thedata as output 1011 for processing by the VI Manager. The AnalysisManager 1002 instructs the VI Manager 112 to render the output 1011 asan interactive visual result list 1086 on the user interface 202 (e.g.in chart form 890, 1208 in the visual representation 18).

Accordingly, in view of the above, and in reference to FIG. 49 d, thespeed Finder 1060 finds occurrences of Entity 24 movement speed based oninterpolated/maximum velocity between two consecutive events 20 and/orat or near a selected location 22. For example, the user can set a rangeand/or maximum velocity 1062,1064 to be found for the data objects 14displayed in the visualization representation 18.

Connection Filtering

Referring to FIG. 49 a, the Connection Filtering Control 1040 allows auser to select one or more data objects 14 and to visualize what theselected data objects 14 are connected to. The Connection FilteringControl 1040 includes object selection functions 1042, 1044 and resultlist 1046. The result list 1046 is formulated by the ConnectionFiltering Manager 1008 and rendered to the visual interface 202 by theVI Manager 112 upon user interaction with the object selection functions1042, 1044. Result list 1046 include results 1048 which represent theconnections 1048 associated with the selected data objects 14. Theresults 1048 are interactive, for example, by clicking on the results1048 to highlight the connections in the visual representation 18. Theprocessing of interactions with the result list 1046 is handled by theVisualization manager 300 which instructs the VI Manager 112 tohighlight the visual images in the visualization representation 18 thatidentify the particular result 1048 that the user has clicked on.

The Connection Filtering Control 1040 also includes an step/connectioncontrol 1043 for refining the number of steps away from the selecteddata object 14 that will retrieved and formulated by the ConnectionFiltering Manager 1008. For example, a user may select entity 24 on thevisual representation 18 and use the step/connection control 1043 toindicate that the user wishes to see what the entity 24 is connected towithin 2 steps. When the user selects object selection function 1043 theinteraction is processed by the Visualization Manager 300.

In an example operation of the module 1040, the Visualization Manager300 instructs the Analysis Manager 1002 to retrieve the subset of dataobjects 14 that corresponds to the search parameter 1043 a. The AnalysisManager 1002 communicates the request to the Connection FilteringManager 1008 which queries the Data Manager 114 with the searchparameter 1043 a. The Data Manager 114 provides the data subset to theConnection Filtering Manager 1008 which formulates the data as output1009 for processing by the VI Manager 112. Finally, the Analysis Manager1002 instructs the VI Manager 112 to render the output 1009 as a visualresult list 1046 on the user interface 202 for viewing and furtherinteraction by the user 202 (e.g. in chart form 890, 1208 in the visualrepresentation 18).

Accordingly, in view of the above, and in reference to FIG. 49 e,Connection Filtering 1040 displays all objects 14,24 connected toselected objects 24, within the selected “x” steps 1043 away. A “step”1048 result is a connection 14 between any two objects 14. For example,an Entity 24 is connected to one of its Events 24 by one step. A link 20between two Entities 24, such as a communication 20, is composed of 4steps, namely Entity_A to call-side event to link event, to receive-sideevent to Entity_B, for example.

Paths Between Objects

Referring to FIG. 49 a, the Paths Between Objects Control 1030 enables auser to select any two data objects 14 to see how they are connected toeach other. The control 1030 includes an object selection function 1032and a result list 1034 that list all the connections 1036 between theselected data objects 14.

In an example operation of the module 1030, when the user selects objectselection function 1032 the interaction is processed by theVisualization Manager 300. The Visualization Manager 300 instructs theAnalysis Manager 1002 to retrieve the subset of data objects 14 thatcorresponds to the search parameters (i.e. the selected data objects14). The Analysis Manager 1002 communicates the request to the PathsBetween Objects Manager 1006 which queries the Data Manager 114 with thesearch parameters. The Data Manager 114 provides the data subset to thePaths Between Objects Manager 1006 which formulates the data as output1007 for processing by the VI Manager 112. Finally, the Analysis Manager1002 instructs the VI Manager 112 to render the output 1007 as a visualresult list 1034 on the user interface 202 for viewing and furtherinteraction by the user (e.g. in chart form 890, 1208 in the visualrepresentation 18).

Accordingly, in view of the above, and in reference to FIG. 49 f, thePaths Between Objects Control 1030 finds Paths 1036 (displayed as pathobjects 14) between 2 selected entities 24 to see how they are connectedand how many steps separate them.

Link Analysis

Referring to FIG. 49 a, the Link Analysis Control 1070 enables the userto select an entity 24 on the visualization representation 18 and tovisualize the degree of separation 1078 for the other entities 24 thatthe selected entity 24 interacts with. The control 1070 includes anentity selection function 1072 and a result list 1074 that is populatedwith results 1076 when the user interacts with entity selection function1072. The results 1076 include a caption 1078 that indicates the degreeof separation between the selected entity 24 and the other entities 24it interacts with.

In an example operation of the module 1070, User interaction with theentity selection function 1072 is processed by the Visualization Manager300. The Visualization Manager 300 instructs the Analysis Manager 1002to retrieve the subset of data objects 14 that corresponds to the searchparameters (i.e. the selected data objects 14). The Analysis Manager1002 communicates the request to the Link Analysis Manager 1016 whichqueries the Data Manager 114 with the search parameters. The DataManager 114 provides the data subset to the Link Analysis Manager 1016which formulates the data as output 1017 for processing by the VIManager 112. Finally, the Analysis Manager 1002 instructs the VI Manager112 to render the output 1017 as a visual result list 1074 on the userinterface 202 for viewing and further interaction by the user (e.g. inchart form 890, 1208 in the visual representation 18).

Accordingly, in view of the above, and in reference to FIG. 49 g, LinkAnalysis Control 1070 determines via selection 1072 the degree ofseparation 1078 for the other entities 24 that the selected entity 24interacts with.

Links Between Entities 24

Referring to FIG. 49 a, the Links Between Entities Control 1050 enablesa user to view all of the links between two or more entities 24. Thecontrol 1050 includes an entity selection function 1052 and a resultlist 1054 that is populated by an interactive list results thatrepresent the links 1056 between the selected entities 24. Selecting twoor more entities 24 on the visual representation 18 can define thesearch parameters.

In an example operation of the module 1052, interactions with the entityselection function 1052 are managed by the Visualization manager 300.The manager 300 instructs the Analysis Manager 1002 to retrieve a subsetof data that corresponds to the search parameters. The Analysis Managercommunicates the instruction to the Links Between Entities Manager whichqueries the Data Manager 114. The Data Manager 114 provides the data setto the Links Between Entities Manager 1012 which formulates the datasubset as output 1013 for processing by the VI Manager 112. The VIManager renders the output 1013 as the visual result list 1056 forviewing and further interaction by the user (e.g. in chart form 890,1208 in the visual representation 18).

Accordingly, in view of the above, and in reference to FIG. 49 h, LinksBetween Entities control 1050 displays all link events (such ascommunications, relationships and other transactions) between 2 or moreselected entities via the entity selection function 1052.

Extensibility

Referring to FIG. 48, the Analysis Module 1000 includes a Plug-inManager 1018 for managing additional analysis tools that a user maydesire to incorporate into the Visualization Tool 12. A user can add newanalysis tools via the user interface 202. The Visualization manager 300processes this type of interaction and instructs the Analysis Manager1000 to create a new analysis tool and to render the visualrepresentation of the new tool to the screen via the VI Manager 112. Itwill be appreciated that any number of analysis tools can be dynamicallyadded to the tool 12 by the user. For example, a user may wish to havean analysis tool to track certain types of events 20 (e.g. such as cashtransactions) or to follow all activity between two locations 22.

Event Aggregation

Referring to FIGS. 1 and 50 a,b, an Event Aggregation Module 1100 isfor, such as but not limited to, summarizing or aggregating event dataobjects 20 in the temporal and/or spatial domain(s), providing thesummarized event data objects 20 to the Visualization Manager 300 whichprocesses translation from event data objects 20 and groups of eventdata elements 27 to the visual representation 18. A plurality of eventdata objects 20 in the visualization representation 18 that match thespecified event aggregation parameter(s) 1101 (e.g. event type(s),specified maximum relative temporal separation between temporallyadjacent events 20, specified maximum relative spatial separationbetween spatially adjacent events 20, specified location(s) 22, and/or acombination thereof, etc.) are aggregated (i.e. collected) into anaggregated event group 27 and then the plurality of individual events 20are replaced in the visual representation 18 by their correspondingaggregated event group 27, thus providing for a potential declutteringof the displayed objects 14 and/or associations 16.

Event 20 types can be such as but not limited to: an elementary event 20having a start time, a location 22, and an associated entity 24 (e.g.can be an event 20 occurring with only, or otherwise attributed to, oneentity 24); a movement event 20 having a start time, an end time, asource place 22, a destination place 22, and at least one associatedentity 24 travelling or otherwise associated with the source/destinationplaces 22; a communication event 20 having a start time, an end time, asource place 22, a destination place 22, and an associated send entity24 (starting/sending the communication 20 from the source place 22) anda receive entity 24 (receiving at the destination place 22 the initiatedcommunication event 20 such as a phone call, email, letter, networkmessage, etc.); a financial transaction event 20 having a start time, anend time, a source place 22, a destination place 22, and an associatedsend entity 24 (starting/sending the transaction from the source place22) and a receive entity 24 (receiving at the destination place 22 theinitiated transaction event 20 such as a banking wire or other fundstransfer operation); and a relationship event 20 having a start time, anend time (optional if still ongoing), and the entity 24 IDs of the twoor more entities 24 in the defined relationship event 20.

Elementary Events 20 are the simplest and most common type of mappingand is the one to use for the two most common types of data 14,16 suchas but not limited to Isolated events where data is composed of distinctevents 20 that each happen independently (e.g. a list of accidents,incidents or readings from sensors that do not move including the timeof the event 20, and position 22 data). Moving object tracks containdata about moving objects or “Entities” 24 that use an object identifieror “selector” column in the data that specifies the object name or ID.Link Events 20 are another type of event 20 such as but not limited to:movement Events 20 where you have travel information that includes bothstart time and location and end time and location in every record (e.g.airline ticket info or shipping travel segment); communication Events 20contain info about 2 objects or Entities 24 communication with eachother with 2 object identifier fields and ideally locations for each(e.g. correspondence, such a letters, email and phone calls data);Financial event 20 having the same information as communications data,but ideally there should be a field for the quantity of the transaction;and Relationship event 20 useful for data that has relationships betweenobjects 14 or Entities 24 (e.g. data about how people are related, orwho they have worked under in an organization).

It is recognised that all of the above discussion on event 20 typescould be used as the parameters 1101 and/or filter 1104 settings.Further, it is recognised that the parameters 1101 are used by thefilters 1104 as dynamically changeable/specified filter 1104 settings.

Referring to FIGS. 3 and 50 a,b, the temporal inter-connectedness ofinformation over time and geography within a single, highly interactive3-D view of the visual representation 18 is beneficial to data analysisof the data in tables 122. However, when the number of event dataobjects 20 increases, techniques for aggregation (or grouping) canbecome important. Many event data objects 14 (e.g. events 20) can becombined into a respective summary or aggregated output 1106 that may berepresented by a single image or icon 27 on the visualizationrepresentation 18. Such outputs 1106 of a plurality of event dataobjects 20 can help make trends in the temporal and spatial domains 400,402 more visible and comparable by the user of the tool 12. Severaltechniques can be implemented to support time/type aggregation of eventdata objects 20, such as, but not limited to grouping within a selectedtemporal context.

As shown in FIG. 50 a, the Event Aggregation Module 1100 has an EventAggregation Manager 1102 that communicates with the VisualizationManager 300 for receiving aggregation parameters 1101 used to formulatethe event aggregated output 1106. The aggregation parameters 1101 can beeither automatic (e.g. pre-defined in the tool 12), manually entered byuser events 109 or a combination thereof. The manager 1102 accesses allpossible event data objects 20 through the Data Manager 114 (related tothe event aggregation parameters 1101, e.g. a time range and/or eventobject 20 types) from the tables 122 (e.g. for example as thosepresently displayed in the visual representation 18, and then appliesevent aggregation filters 1104 (based on the aggregation parameters1101) for generating the output 1106. The VI Manager 112 receives theoutput 1106 from the Event Aggregation Manager 1102 and renders theoutput to the screen as the visual representation 18.

The filters 1104 act to organize and aggregate the event data objects 10in the temporal domain according to the instructions provided by theEvent Aggregation Manager 1102. For example, the Event AggregationManager could request that the filters 1104 summarize all events 20(further alternatively or in addition to of a selected event type) thatoccur within 30 minutes of each other (e.g. a maximum temporalseparation between displayed icons representing event objects 20). Oncethe event data objects 14 are matched by the filters 1104 as matchingthe specified/selected aggregation parameters 1101, the aggregated eventdata 27 (i.e. a reduced number of displayed event data objects ascompared to the previously displayed un-aggregated event objects) issummarized as the output 1106. The Event Aggregation Manager 1102communicates the output 1106 to the VI Manager 112 to rendering as thevisual representation 18. It is recognized that the content of therepresentation 18 is modified to display the output 1106 to the user ofthe tool 12, according to the aggregation parameters 1101.

As a further example, a user may choose to summarize all events 20 thatoccur within 30 minutes of each other at a specific location 22 (e.g.Rome). To accomplish this, the user indicates their preferences toaggregate the event data 20 according to temporal and local proximity byuse of the controls 306. The Visualization Manager 300 communicates theaggregation parameters to the Event Aggregation Manager 1102 in order tofilter the event data objects 20 for display on the visualizationrepresentation 18. The Event Aggregation Manager uses the filters 1104to filter the data from the tables 122 based on a spatial proximity andtemporal properties. The output 1106 is rendered to the screen by the VIManager 112.

The aggregated output 1106 may be represented on the visualizationrepresentation 18 in one of several ways. For example, aggregations of aplurality of event data objects 20 that occur in the specifiedaggregation time period (e.g. 30 minutes as specified using a start andend time (e.g. specified time range) selected by the user may berepresented by a single icon 410 (or a relatively reduced number oficons 410) on one or more timelines 422. It is appreciated that a usermay interact with the icon to determine if the icon 410 represents asingle event 20 or multiple events 20 that have been aggregated as event27. When the user places the mouse cursor 713 over the visual element oricon 410, for example, pre-determined information about the aggregationdetails of the event data objects 27 that are represented by the icon410 may be displayed. Information related to each of the event dataobjects 20 that have been aggregated may also appear. The user cancancel the aggregation 27 that has been applied by interacting withcontrols 306. The visualization manager 300 then communicates theinstruction to the VI Manager 112 which recreates the visualizationrepresentation with no aggregation parameters displayed.

Referring to FIGS. 50 a,b, the filters 1104 combine events 20 that areclose to each other and display them as one or more aggregated events 27in the representation 18. The resulting aggregated events 27 can have a“size” value (e.g. displayed physical extent of event 20 icon) thatcorresponds to the number of individual events 20 that were aggregatedinto the aggregated event 27. Filters 1104 are good for reducing thenumber of events 20 when detail is not important, or when attempting todisplay very large sets of data, for example. There can be a number oftypes of filters, for example filters such as but not limited to:Identical Events such as duplicate events (e.g. only a representativeone of the duplicated events 20 is displayed); Events close in specifiedtime and/or spatial position, such that displays of an aggregated event27 are used to represent a cluster of events 20 that are closely related(e.g. satisfy or otherwise match the specified parameters 1101). Forexample, a Filter Properties panel (not shown) provides for the user to:set filter 1104 tolerances by setting the distance in time and/or space1101 within which events 20 will be aggregated; and inhibit aggregationof events 20 that have specified 1101 different labels, comments, oruser defined attributes. For example, differences can be ignored, andany events 20 within range of the specified parameters 1101 areaggregated unless they belong to different Entity 24 tracks.

Time Charts 890

Referring to FIGS. 46 a, 46 f, 47, a chart Module 860 is for, such asbut not limited to, summarizing data elements 874 (e.g. selected dataobjects 14, 16 such as but not limited to entities 24, locations 22,events 20, associations 16) in time periods of the focus range 844 (orat least a selected portion of the focus range 844) and providing thesummarized data elements (e.g. entries 891) as a chart output 866 to theVI Manager 112 for rendering to the visual interface 202. Shown, byexample only, is a table format for the entries 891 correlated usingchart divisions of columns for each selected data element 874 (e.g.entities 24) and rows corresponding to each time indication 847, 856 ofthe respective bar 840, 850. In the present example, the rows correspondto the indications 856.

The chart Module 860 can include a chart Manager 862 and a Chart Manager864. The module 860 is in communication with the Visualization manager300, for example. User events 109 are generated as a user interacts withthe data element selector 872 (see FIG. 46 a) and the user interface 202and are directed to the Visualization manager 300. The Visualizationmanager 300 communicates the summarized data elements 874 to the chartModule 860 and instructs the module 860 to generate a chart/time tableas chart output 866. The chart Module 860 can interact with the DataManager 114. Specifically, the chart Manager 862 can instruct the DataManager 114 to retrieve a data set 861 that includes the data elements874 of the visualization representation 18 that are within (e.g. satisfyor otherwise match) the specified focus range 844 (or portion thereof)and the selections 872. The Data manager 114 directs the data set 861 tothe chart Manager 862 for processing and for rendering on the visualinterface 202 as the entries 891. The chart Manager forms the output 866and can instruct the VI Manager 112 to render the chart output 866 tothe user interface 202 in the form of a chart 890.

The chart Module 860 can also include a Chart Manager 864 that producesthe chart output 866 depending on the preferences of the user. As shownin FIG. 46 a, the user can set the time period by interacting with atime period selector (e.g. specifying the start and end time 841,843).When the user selects the time period, the Visualization manager 300 caninstruct the chart Manager 862 to create a chart 890 and to render thechart 890 (i.e. including the entries 891 and associated selected dataelements 874—e.g. as specified entities 24) to the visual representation18 on the user interface 202. The chart Manager 862 can communicate therequest to the Chart Manger 864 which generates chart output 866 whichis directed to the VI Manager 112 for rendering to the visual interface202 as the chart 890.

Referring to FIGS. 46 a, 46 f, the data elements selector 872 includes adrop-down list 873 which contains a list of all data elements 14,16 thatreside in the tables 122 (see FIG. 3) for the objects 14 and/orassociations 16 that have temporal attributes in the defined limits841,843. A user may wish, for example, to create a chart 890 thattabulates the occurrence (e.g. count, frequency) of entities 24 having atemporal attribute in the focus range 844. As shown, the chart 890includes columns 880 that correspond to the selected data element 874(i.e. entities 875 a-875 f in FIG. 46 a). The occurrence of each of thedata elements 874 at the focus indicators 856 is visualized with a glyphor icon 882 placed in the row 884 that corresponds to the focusindicator 856 (i.e. each of the icons 882 is positioned in the chart 890aligned with the focus indicator that represents their associatedtemporal attribute—e.g. point in time of the temporal reference domain).The glyph or icon 882 may have display properties (e.g. colour, size,shape, etc.) that represent(s) the nature or importance of the dataelements 874 (or any other characteristic). It is appreciated that theglyph or icon 882 can be rendered to the screen by the VI Manager 112 asdescribed above.

It may be desirous for the user to summarize the chart 890 in “buckets”(e.g. specified periods) of time referred to as a summary time period876 (e.g. a selected summary portion 876 of the total focus bar timerange 844 (i.e. between the limits 841,843). For example, a user maywish to quickly view the occurrence of data elements 874 at the summarytime period 876 of an hour, a day, a month or any other period of timethat is within the focus range 844. When a summary time period 876 isselected using the summary time period selector 877, the manager 300instructs the chart Manager 862 to create a Chart Output 866 which isrendered to the screen as a Summary Chart 890 a,b by the VI Manager 112.The Summary Chart 890 a,b can be a bar chart that includes bars 891(e.g. summary entries) representing each summary time period 876. Thelength of the bars 891 in the vertical direction is a representation ofthe number of occurrences of data elements 874 at each summary timeperiod 876. It will be appreciated that the Summary Chart 890 a,b may bevisualized as a bar chart as shown in FIG. 46 a or may be visualized asa pie chart, a continuous plotting of data elements 874 across time, orany other suitable chart. Further, it is recognized that the chart 890can have chart divisions other than the rows and columns as described,for example alternative chart divisions such as pie wedges of a piechart.

The Summary Chart 890 a,b is capable of summarizing the occurrence ofall the data elements 874 in each summary time period or only for aspecific data element 874. For example, in FIG. 46 a, the chart 890summarizes the occurrence of entities 875 a-875 f. A user can select thesummary time period 876 using the summary time period selector 877. Ifthe user wishes to create the Summary Chart 890 a for only 875 a, theuser can select the column 880 that represents entity 875 a. Uponselecting the row 884 a, the user interface 202 generates a user event109 and directs the user event 109 to the Visualization manager 300. TheVisualization manager 300 instructs the chart Manager 862 to createchart output 866 that corresponds to the occurrence of entity 875 a inthe selected summary the time period 876 for rendering to the userinterface 202 by the VI Manager 112, as summary chart 890 a.

A user may also create several Summary Charts 890 a, 890 bsimultaneously that represent the occurrence of each entity 875 a-f indifferent time periods (i.e. a plurality of selected time periods thatrepresent subdivisions of the time range of the focus range 844. Forexample by interacting with the controls 872 and 877, a user may chooseto create a summary chart 890 a for the period between 856 c and 856 d,and another summary chart 890 b for the period between 856 a and 856 b.As before, each of the bars 891 represent the selected data elements 874and the height of the bars 891 represent the occurrence of each dataelement 874 in the time period adjacent the summary charts 890 a, 890 bon the focus bar 850, for example.

Count Charts

Referring to FIGS. 51, 52, the Count Chart Module 1200 is for, such asbut not limited to, summarizing the count of data objects 14 in anoutput 1206, providing output 1206 to the VI Manager for display as aCount Chart 1208 (e.g. an example of the summary chart 890 a,b, see FIG.46 a). A Count Chart 1208 is a bar chart in which each bar 1212corresponds to a data object 14, and the length of each bar representsthe count of another data element that is associated with the dataobject 14. The Count Chart Module 1200 also provides another interactivemechanism for allowing the user to control the data objects 14 that arevisualized on the visual representation 18 and throughout the userinterface 202, e.g. selection of the particular data element(s) 14 (e.g.a selected bar(s) 1212) results in the corresponding visual elementsthat represent the selected data element(s) being only displayed (orotherwise highlighted/emphasized in the visual representation 18.

As shown, the Count Chart Module 1200 has a Count Chart manager 1202that communicates with the Visualization Manager 300 for receivinginstructions (i.e. criteria for creating the Count Chart 1208 such asbut not limited to the selected data object(s) 14 that will correspondto the bar(s) 1212) used to formulate the output 1206. The Count Chartcriteria can either be pre-loaded in the tool 12, manually entered bythe user via user events 109 or a combination thereof. Upon userinteractions with count control 1240 (see FIG. 52), e.g. an example ofthe interface controls 306, the Visualization Manager 300 communicateswith the Count Chart manager 1202 and instructs the manager 1202 togenerate a Count Chart 1208. The Count Chart manager 1202 createsfilters 1204 based on the search criteria and retrieves a result dataset from the Data Manager 114. It is appreciated that the result dataset represents the set of data objects 14 that are within/match orotherwise satisfy the defined search criteria.

An example Count Chart control 1240 is illustrated in FIG. 52. Thecontrol 1240 includes a data element selector 1242, a distance selector1244, a count selector 1246, a result Count Chart 1208 and toggles 1210.To create a Count Chart 1208, a user selects the data element using thedata elements selector 1202 that the user wishes to chart. As shown, theselected data type is represented by bars 1212 in the Count Chart 1208.A user also selects the data type that the user wishes to count inrelation to the selected data element. In FIG. 52, the user has chosento count events 20 that are related to entities 24. As described above,the length of bars 1212 represents the count of data types in relationto the selected data element 1242. In FIG. 52, it is apparent that the“Taxi 2234” is related to far more events than “Dispatch A”. The CountChart 1208 also includes a count 1214 so the user can quantify the countrepresented by the length of bars 1212.

The Count Chart control 1240 also includes toggles 1210 for selectingand deselecting the selected data elements that the user wishes tovisualize in the Count Chart 1208. For example, if the user deselects(i.e. unchecks) the toggle 1210 that is beside the entity labelled as“Victim” the row indicated by 1212 a will disappear from the Count Chart1208. As is described in more detail below, interactions with the CountChart 1208 may affect the visual contents of the visual representation18 and the time table 870.

User interactions with the control 1240 are processed by theVisualization Manager 300. The manager 300 communicates with the CountChart manager and instructs the Count Chart manager 1202 to create a newCount Chart 1208. The Count Chart manager creates filters 1204 whichrepresent the user inputs to controls 1202, 1204 and 1206. The CountChart manager retrieves a data set from the Data Manager based on thesearch criteria of the filter 1204. Finally, the Count Chart manager1202 formulates the data set as an output 1206 and instructs the VIManager to create and draw the visual Count Chart 1208.

It is to be appreciated that a user may choose to generate a Count Chart1208 for any data element in the table 122 available between the definedlimits 841,843. For example, a user may set the selected data element toa location data object 22 (through the control 1242) and choose to countall events 20 that occur at the that location 22 (through the control1246)

It is to be appreciated that the Count Chart Control 1250 is coupled ina dynamic relationship to the contents of visualization representation18, the contents of the timeline bar 840 and the contents of the timetable 890. The Visualization Manager 300 monitors user interaction inthe form of user events with each of the above and coordinates theresponse of the other modules. For example, increasing the focus range844 on the timeline bar 840 increases the number of data objects 14 thatmay be represented as images on the visualization representation 18. Tothis end, the Visualization Manager 300 generates and/or updates sprites308 and communicates the sprites to the VI Manager 112 for rendering anew visualization representation 18 to the screen. Likewise, theVisualization Manager 300 communicates with the Count Chart manager 1202and instructs the Count Chart manager 1202 to generate a new Count Chart1208 to visualize and count the additional data objects 14 that are nowwithin the amended focus range 844. Likewise, interaction with the CountChart control 1250 may affect the visual images represented on thevisualization representation 18 and the time table 890. As an example,when a user deselects a data object 14 on the Count Chart 1208 usingtoggles 1210, the Visualization Manager 300 instructs the Count Chartmanager 1202 to create a new output 1206 and to render the new output inthe form of a Count Chart 1208 via the VI Manager 112. The VisualizationManager 300 deletes sprites 308 that correspond to the deselected dataelement 14 and instructs the VI Manager to create and display a newvisualization representation 18. The deselected data element 14 is notrepresented on the visualization representation 18 as the sprites 308that facilitate rendering of the images have been deleted by theVisualization Manager 300. As another example, if a user desires toselect a group of data objects 14 on the visualization representation 18using the mouse pointer 713, the Visualization Manager 300 communicateswith the Count Chart manager 1202 and instructs the manager 1202 tocreate a new Count Chart that represents the data objects 14 that havebeen selected, for example displayed as a chart annotation 21 describedbelow. Likewise, the Visualization Manager 300 instructs the Time Tablemanager 862 to create a new time table 890 for the selected data objects14.

Accordingly, in view of the above, the count chart 1208 can be used as acount summary for representing the count of the entries 891 in theselected chart 890 and/or sub-charts 890 a,b, such that the count chart1208 generation is associated with the entries 891, time indicators847,856 (e.g. specific and/or range(s) thereof), and selected dataobjects 874 done for the chart 890 and/or sub-charts 890 a,b.

Annotations 21

Annotations 21 in Geography and Time can be represented as manuallyplaced lines or other shapes (e.g. pen/pencil strokes, charts, etc.) canbe placed on the visual representation 18 by an operator of the tool 12and used to annotate elements of interest with such as but not limitedto arrows, circles and freeform markings and summary chart details. Someexamples are shown in FIGS. 53-58, discussed below. The annotations 21are graphics that can be selected and positioned on the visualrepresentation 18 for use in highlighting or otherwise providing addedmeaning/description to the patterns uncovered by the user in the dataobjects 14 and associated associations 16. These annotations 21 arelocated in geography (e.g. spatial domain 400) and time (e.g. temporaldomain 422) and so can appear and disappear on the visual representation18 as geographic and time contexts are navigated through the user inputevents 109 and therefore displayed as temporal and/or spatial dependentcontent in the visual representation 18.

For example, one application of the tool 12 is in criminal analysis bythe “information producer”. An investigator, such as a police officer orother law enforcement, could use the tool 12 to review an interactivelog of events 20 gathered during the course of long-term investigations.Existing reports and query results can be combined with user input data109, assertions and hypotheses, for example using the annotations 21.Further, the tool 12 could also have a report generation module thatsaves a JPG format screenshot (or other picture format), with a titleand description (optional—for example entered by the user) included inthe screenshot image, of the visual representation 18 displayed on thevisual interface 202 (see FIG. 1). For example, the screenshot imagecould include all displayed visual elements 410,412, including anyannotations 21 or other user generated analysis related to the displayedvisual representation 18, as selected or otherwise specified by theuser.

The visualization tool 12 also has associated modules (e.g. anannotation module such as the visualization manager 300) formanipulating the properties of the created/defined annotations 21 aswell as for creating/defining new annotations 21. For example, the usercan use the visualization tool 12 to lock an annotation 21 so that italways stays visible (the location and/or orientation of the lockedannotation 21 changes) in the visualization representation 18 as theuser manipulates the viewing angle/zoom (e.g. panning) of the displayedobjects 14 and/or annotations 16. The locked annotation 21 can remainvisible (e.g. preferably displayed in the visual representation 18 in arelatively sparsely populated area (e.g. containing relatively fewerobjects 16 and/or associations 16 as compared to other regions of thevisual representation 18) as long as the underlying objects 14 and/orannotations 16 it is attached to are in the displayed visualrepresentation 18.

Further, the visualization tool 12 can be used to pin an annotation sothat it stays in the same position on the screen no matter the tilt orrotation (i.e. the annotation 21 remains in a predefined location and/ororientation of the visual representation 18, regardless of thetilt/rotation/zoom manipulations of the visual representation 18 by theuser. It is recognized that the annotations 21 visible in the visualrepresentation 18 can be dependent on the selected time range and/orspatial range for display of their related data objects 14 (e.g. anannotation 21 attached to a selected data object 14 would only bedisplayed in the visualization representation 18 if the associated dataobject 14—e.g. entity 24—has the parameter(s) of a time and/or alocation that is in the displayed time and/or spatial ranges of thevisualization representation 18).

Accordingly, annotations 21 can be defined as graphics used to highlightor describe patterns found by the user in the data objects 14 and/orassociations 16. Annotations can be used for communicating findings inthe data for presentation to others, or to help the remembering ofperformed analysis. Annotations 21 can be applied to events or otherobjects 14 and/or association 16 (e.g. selected grouping(s) thereof) inthe visual representation 18, and the annotations 21 remain attached(e.g. via link 21 a) to their data 14,16 as the user navigates in thevisual representation 18. The annotations 21 can be locked so that theyare always visible, and they can also be saved as part of a snapshot(e.g. saved in a report) so that they can be recalled when needed.

Types of Annotations 21

Referring to FIGS. 53-58, the following example annotation 21 optionsare available from an Annotation Toolbar dropdown button (e.g. interfacecontrols 306—see FIG. 5), for example, such as but not limited to: Groupcontrol 306 a that draws a circle 21 a around selected events 20;Callout control 306 b that annotates an event 20 or group of objects 14with a descriptive text 21 b; Chart Summary of information control 306 cthat is annotated to an event 20 or group of objects 14 displayed as abar graph 21 c (or other graph type—e.g. pie chart); Line control 306 dthat connects selected elementary events 20 with a (e.g. dashed) labeledline 21 d; a ruler control 306 e that produces an annotation ruler 21 e(or other line type) that can calculate/display the time, distance andspeed of an entity 20 based on the time and distance between selectedevents/objects 14; and a Symbols control 306 f that represents a set ofpredefined icons 21 f used to annotate selected object(s) 14 and/orassociation(s) 16.

The Chart Annotation 21 c is used to annotate selected entities 24 withassociated events 20 (e.g. number and/or type of event(s) 20) in chartssimilar to the one found under the Charts tab provided by the tool 12.Like all annotations 21, the Chart Annotation 21 c can be captured in asnapshot used for Generating Reports. The following charting categoriesare available in the Chart dropdown within the Chart Annotation CategoryFunction 306 c, for example such as but not limited to: Entities 24associated with the event 20; Label Contents of event Label Field; SizeValues in the size field; Color Event display color; Places 22associated with the event 20; Icon File Contents of the event IconField; Data File Contents of the event Data Field; selected times (e.g.time ranges) for events 20 to be included in the chart 21 c (e.g. Hourof Day Event time by the hour it falls into of the 24 hour clock, Day ofWeek Event time by day of week it falls into of the 7 day week(Monday-Sunday), Month of Year Event time by month of year it falls intoof the 12 month year; Time by Hour Event start time by specific hourunique to each particular day and month, Time by Month Event start timeby specific month unique to each particular year, Time by Year Eventstart time by year); and User Data Fields. The visualization tool 12 canalso provide for charting 21 c by any additional event 20 attributesincluded as additional columns in the chart 21 c when sent from Excel orother data providers.

A further annotation 21 type is Group Annotation 21 a, see FIG. 55, tohighlight a specific set of object(s) 14 and/or association(s) 16 thatplay a determined important part of the analysis. Like all annotations21, the Group Annotation 21 a can be captured in a snapshot used forGenerating reports. A further annotation 21 type is the Line Annotation21 d, see FIG. 56, to connect related events 20 with a labeled line 21d. Like all annotations 21, the Line Annotation 21 d can be captured ina snapshot used for Generating Reports. A further annotation 21 type isthe Ruler Annotation 21 e, see FIG. 57, used to show the distance, timeand velocity between two events. Like all annotations 21, the RulerAnnotation 21 e can be captured in a snapshot used for GeneratingReports. A further annotation 21 type is the Symbols Annotation 21 f,see FIG. 58, used to mark significant selected object(s) 14 and/orassociation(s) 16 with a relevant symbol 21 f selected by the user ofthe tool 12. Like all annotations 21, the Symbol Annotation 21 f can becaptured in a snapshot used for Generating Reports. Accordingly, chart21 c and ruler 21 e annotations can be considered as having analyticalfunctions associated with them that displays calculations of selectedcharacteristics of the data objects 14 and associations 16 that theannotations are coupled to on the visual representation 18.

Working with Annotations 21 (e.g. Right Clicking on the Annotation 21)

The following menu items can appear on the user interface 202 when thedisplayed annotation 21 is selected in the visual representation 18,such as but not limited to: Move Closer—zooms annotation 21 into centerof view, or selected items in both space and time; Move Further—zoomsannotation 21 out from center of view, or selected items in both spaceand time; Fit annotation 21 to a set of presets used to zoom in on datain the Space-Time Viewer; Fit Data Extent—zooms annotation 21 to thetime-space extents of all data loaded into tables 122, and resets todefault view position; Fit both time and space of annotation 21 toselected; Fit time of annotation 21 to selected; Fit space of annotation21 to selected; Delete Selected annotation 21 from representation 18;Edit Annotation such that the tool 12 edits information annotated inCallouts and Charts annotations 21; Remove Annotation—deletesannotations 21; Locked—protects annotations 21 from being deleted by theRemove Annotation Button; Pinned—fixes the position of an annotation 21in the representation 21 such that selected annotations 21 will nolonger move when associated object data is moved in the representation18; and Object Properties—opens the Object Properties Panel of theannotation 21 which contains additional information about the annotation21 selected in fields that can be edited.

Example Operation of the Tool 12

Referring to FIG. 46 a and related figures, the method implemented bythe various modules of the tool 12 for configuring the presentation of aplurality of presentation elements in a visual representation on a userinterface, the presentation elements having both temporal and spatialparameters, comprises at least some of the steps of: 1) defining a timebar with a time scale having time indicators as subdivisions of the timescale and having a first global temporal limit and a second temporalglobal limit of the time scale for defining a temporal domain of thepresentation elements, 2) defining a focus range of the time bar suchthat the focus range has a first local temporal limit and a second localtemporal limit wherein the first local temporal limit is greater than orequal to the first global temporal limit and the second local temporallimit is less than or equal to the second global temporal limit; 3)defining a focus bar having a focus time scale having focus timeindicators as subdivisions of the focus time scale and having the firstand second local temporal limits as the extents of the focus time scale,such that the focus time scale is an expansion of the time scale; and 4)displaying a set of presentation elements selected from the plurality ofpresentation elements based on the respective temporal parameter of eachof the set of presentation elements is within the first and second localtemporal limits.

Further steps are: 5) defining the focus range as a focus sliderconfigured for displacement along the time scale between the first andsecond global limits resulting in updating of the first and second locallimits, such that the focus range remains constant during thedisplacement of the focus slider; wherein during the displacement theset of presentation elements is updated from the plurality ofpresentation elements based on the respective temporal parameter of eachof the updated set of presentation elements is within the updated firstand second local temporal limits; 6) generating a chart for positioningin relation to the focus bar, the chart having a plurality of entriescorresponding to those presentation elements selected from the pluralityof presentation elements matching their temporal parameter to be withinthe focus range, such that the entries are relatively positioned to oneanother in the chart in ordered relation to their respective temporalposition in the focus time scale using spacing between the entriescorresponding with that spacing of the focus time indicators of thefocus time scale; 7) wherein the chart is positioned adjacent to thefocus bar such that the entries are lined up with their correspondingfocus time indicators; 8) wherein the matching presentation elementshave a spatial parameter matching a specified spatial search criterion;9) wherein the chart is a summary chart generated for a specified dataelement selected from the matching presentation elements; 10) whereinthe summary chary is positioned in the visual representation in definedassociation to the chart; 11) wherein the chart includes a series ofcolumns and rows or other divisions; 12) using an analysis operator ingenerating the chart such that the plurality of entries also satisfy atleast one analysis criterion of the analysis operator; 13) wherein theanalysis operator is selected from analysis means selected from thegroup comprising: meeting finder; gap finder; speed finder; connectionfiltering; path between presentation elements; link analysis; and linkbetween entities.

The invention claimed is:
 1. A method for configuring the presentationof a plurality of presentation elements in a visual representation on auser interface, the plurality of presentation elements having bothtemporal and spatial parameters, the method comprising the steps of:defining a time bar with a time scale having time indicators assubdivisions representing a unit of measure of the time scale and havinga first global temporal limit and a second temporal global limit of thetime scale for defining a total temporal domain of the plurality ofpresentation elements; defining a focus range of the time bar such thatthe focus range has a first local temporal limit and a second localtemporal limit wherein the first local temporal limit is greater than orequal to the first global temporal limit and the second local temporallimit is less than or equal to the second global temporal limit, thefocus range being a subset of the total temporal domain; defining afocus bar having a focus time scale having focus time indicators assubdivisions representing a unit of measure of the focus time scale andthe focus bar being in a dynamically coupled relationship with the timebar by having the first and second local temporal limits of the time baras the extents of the focus time scale, wherein the focus time scale isan expansion of the time scale by a unit distance on the user interfacebetween the focus time indicators being greater than a unit distance onthe user interface between the time indicators; displaying a subset ofpresentation elements selected from the plurality of presentationelements based on the respective temporal parameter of each of thesubset of presentation elements being within the first and second localtemporal limits; manipulating the focus range by displacement along thetime bar between the first and second global limits to result inupdating of the first and second local limits being the extents of thefocus time scale, wherein the focus range remains constant during thedisplacement; wherein during the displacement the display of the subsetof presentation elements is animated over time and geography by updatingfrom the plurality of presentation elements based on the respectivetemporal parameter of each of the updated subset of presentationelements is within the updated first and second local temporal limits.2. The method of claim 1 further comprising the step of generating achart for positioning in relation to the focus bar, the chart having aplurality of entries corresponding to the subset of presentationelements selected from the plurality of presentation elements matchingtheir temporal parameter to be within the focus range, wherein theentries are relatively positioned to one another in the chart in orderedrelation to their respective temporal position in the focus time scaleusing spacing between the entries corresponding with that spacing of thefocus time indicators of the focus time scale.
 3. The method of claim 2,wherein the chart is positioned adjacent to the focus bar wherein theentries are lined up with their corresponding focus time indicators. 4.The method of claim 2, wherein the chart is a summary chart generatedfor a specified data element selected from the matching presentationelements.
 5. The method of claim 4, wherein the summary chart ispositioned in the visual representation in defined association to thechart.
 6. The method of claim 2, wherein the chart includes a series ofcolumns and rows.
 7. The method of claim 2 further comprising the stepof using an analysis operator in generating the chart wherein theplurality of entries also satisfy at least one analysis criterion of theanalysis operator.
 8. The method of 7, wherein the analysis operator isselected from analysis means selected from the group comprising: meetingfinder; gap finder; speed finder; connection filtering; path betweenpresentation elements; link analysis; and link between entities.
 9. Themethod of claim 1 wherein the matching presentation elements have aspatial parameter matching a specified spatial search criterion.
 10. Asystem for configuring the presentation of a plurality of presentationelements in a visual representation on a user interface, the pluralityof presentation elements having both temporal and spatial parameters,the system comprising a memory device having stored instructions thereonfor execution by a processor to: define a time bar with a time scalehaving time indicators as subdivisions representing a unit of measure ofthe time scale and having a first global temporal limit and a secondtemporal global limit of the time scale for defining a total temporaldomain of the plurality of presentation elements define a focus range ofthe time bar having a first local temporal limit and a second localtemporal limit wherein the first local temporal limit is greater than orequal to the first global temporal limit and the second local temporallimit is less than or equal to the second global temporal limit, thefocus range being a subset of the total temporal domain; define a focusbar having a focus time scale having focus time indicators assubdivisions representing a unit of measure of the focus time scale andthe focus bar being in a dynamically coupled relationship with the timebar by having the first and second local temporal limits of the time baras the extents of the focus time scale, wherein the focus time scale isan expansion of the time scale by a unit distance on the user interfacebetween the focus time indicators being greater than a unit distance onthe user interface between the time indicators; display a subset ofpresentation elements selected from the plurality of presentationelements based on the respective temporal parameter of each of thesubset of presentation elements being within the first and second localtemporal limits; manipulate the focus range by displacement along thetime bar between the first and second global limits to result inupdating of the first and second local limits being the extents of thefocus time scale, wherein the focus range remains constant during thedisplacement; wherein during the displacement the display of the subsetof presentation elements is animated over time and geography by updatingfrom the plurality of presentation elements based on the respectivetemporal parameter of each of the updated subset of presentationelements is within the updated first and second local temporal limits.11. The system of claim 10 further comprising generate a chart forpositioning in relation to the focus bar, the chart having a pluralityof entries corresponding to the subset of presentation elements selectedfrom the plurality of presentation elements matching their temporalparameter to be within the focus range, wherein the entries arerelatively positioned to one another in the chart in ordered relation totheir respective temporal position in the focus time scale using spacingbetween the entries corresponding with that spacing of the focus timeindicators of the focus time scale.
 12. The system of claim 11, whereinthe chart is for positioning adjacent to the focus bar wherein theentries are lined up with their corresponding focus time indicators. 13.The system of claim 11, wherein the chart is a summary chart generatedfor a specified data element selected from the matching presentationelements.
 14. The system of claim 13, wherein the summary chary is forpositioning in the visual representation in defined association to thechart.
 15. The system of claim 11, wherein the chart includes a seriesof columns and rows.
 16. The system of claim 10 wherein the matchingpresentation elements have a spatial parameter matching a specifiedspatial search criterion.
 17. The system of claim 10 further comprisinggenerate the chart wherein the plurality of entries also satisfy atleast one analysis criterion of the analysis operator.
 18. The system ofclaim 17, wherein an analysis operator used in said generate the chartis selected from the group comprising: meeting finder; gap finder; speedfinder; connection filtering; path between presentation elements; linkanalysis; and link between entities.